My gmail account was hijacked apparently. Many of the addresses on my address list received spam from me at 9:00 am EST 4/24/2010. This makes the issue of email identity theft personal for me. It is an issue that needs to be addressed before it effects you.
I've changed passwords, etc. But once they have an email list, it gets shared among spammers and I cannot stop them from using the email addresses forever. Damn.
What most likely happened is that a sight was hacked where i registered with the same password as gmail. Shame on me Or, I shared my address list on a social networking site that was hacked, shame on them.
If your email was in my address book beware of links presumably sent by me but not signed "Jim". They are not from me really, they are spoofed. Sorry folks.
Now I understand why emails clearly not spam from some members are flagged as spam in groups I moderate, and why email from some people often gets blocked as spam. I had wondered if they had been spammers in their past that they were being punished now. It makes more sense that they had been victims of email identity theft as I now am.
I ban members from my groups who enter spam, and report spam when I receive it. Unwittingly I have been victimizing the victims of email identity theft. As I judged, now I am judged. If the spammers send a lot of email from my account my electronic world will get smaller and smaller. The victims of email identity theft will become isolated and not contribute what they otherwise could to society.
I am still trying to notify everyone in my email address book, but there are too many addresses and gmail blocks me from sending email for 24 hours when I hit the limit. I can't even defend myself.
It is impossible for any national government to control email identity theft. China. a most repressive nation is where must of the spam originates or is relayed. It is a good thing the internet is an open bottom up protocol. Only name services are controlled centrally. The problem is not the freedom of the internet, the problem is that 99.99% of us are not using digitally signed email.
This will not change until email clients people use to send email validate and encourage using digital signatures, and that will not happen until people choose to only use clients that do. A digital signature uses cryptographic techniques to provide verifiable proof of authorship. This has not happened because standards need to be accepted to accept digital signatures in emails that are backwards compatible with the billions of legacy clients. There is no accepted standard means of including a digital signature in a plain text unencrypted ordinary email. Any suggestions?
The problem is the transition. If there was a standard way to digitally sign a plain text email in the header or the body, any email client that already had your public key, or could get it, could flag spoofed emails as not really from you. Email programs that did not support the signitures would still work fine but with no identity check. Email providers and mailing list and forwarding programs could have keys for their email addresses and support this
The use of a service, one time, to create and serve your keys, and an email client that supports it, is all that is needed.
If one big email provider, like google, supported this, I think it would give them a market advantage, and every client would follow along. The threat of email identity theft and spam generally would be greatly reduced.
Jim
Links to more info:
http://mail.google.com/support/bin/answer.py?hl=en&answer=50200
http://answers.google.com/answers/threadview/id/14096.html
http://www.scamdex.com/
http://en.wikipedia.org/wiki/Digital_signature
http://en.wikipedia.org/wiki/Pretty_Good_Privacy
Sunday, April 25, 2010
Wednesday, April 14, 2010
Remembering GrampScarver
Remembering GrampScarver
[My memorial service testimonial plus added missing sentiments expressed by others, afterthoughts, etc.]The financial crises has left me out of regular employment and I have been thus blessed with the wonderful opportunity to be with mom and dad through our challenges of the last year. Dad lived well and died well. It was almost a year ago when he announced his intention to die. He told us all he had lived a good life and done everything he wanted to do. Near the end, he blamed me that he lived too long for taking too good care of him. I feel, he was not going anywhere until he was convinced my mom would be okay.
In the end he said he was making no more decisions. He was delegating to me the hard job of making all the decisions for him. I said "dont be silly, taking care of you and mom is easy and I am happy to do it. It is a blessing for us all that I can be here now." I am beginning to see the gravity of the responsibility I accepted that is only increased in his death. To do his will, I must remember all that he taught us by his words and by how he lived.
He was the independent individual in many ways. He was always ahead of his time technologically. We were years ahead of the croud in having a rotary lawn mower, video game, microwave oven, digital watch, calculator, computer, etc. He never bought what was the current fad, he got what he wanted that nobody else had.
You would not call him gregarious as he shied from social activities. He seemed a bit cold and distant on first impression. Yet everyone liked him and he was most always pleasant, helpful, knowledgeable, and intelligent. Everyone respected him. Many sought his council. He was an avid reader.
Delegate everything. His business ran like clockwork without the necessity of his involvement. Instead he spent time learning, strategic planning, and working alongside every employee every year. His employees, for the most part, loved him, and too good care of business for him.
Every problem is an opportunity, the challenge is finding it. There are no problems, only solutions.
When I wrapped the new family car around a tree, and said, "Gee dad I'm sorry." he said, "No problem, things are not important, people are." That material things are not important to me is a great gift he has given me. I do not suffer, as many do, over material things. Money was never a problem, so it was not important. He was frugal though he did not need to be, and never succumbed to consumerism.
I grew up thinking he knew everything. Sitting on his knee he taught me everything from psychology and political theory to quantum and space time physics. It was not until I was 35 that I asked him a question and he was not familiar with the prevalent views and issues and said simply, I don't know, leaving me in a state of shock.
Whenever we said, "it was an accident", he would interject, "There is no such an accident, everything happens for a reason. "
Only you are responsible for what happens to you.
He had a most unconventional wisdom and was a master of reverse psychology. He gave old adages a Twainian twist that was uniquely Franklin.
Spend your money, don't save it. Any idiot can put money in the bank, the real challenge is to spend it well.
Never set an attainable goal for your life, since once you attain it, your purpose in life is lost.
If you are only willing to try 99 times, don't bother trying
If you try to do two things, you will not succeed at neither. If you do one thing, nothing is impossible.
The most intelligent person is the one who makes all possible mistakes first.
It is better to act, and be wrong, than to not act.
When he attained his financial goals in business he no longer had any interest in business and began a new technical career from the bottom. He told us to never set an attainable goal for our life, since once attained, purpose in life would be lost.
Never concern yourself with what you cannot do, only consideration of what you can do is worth while. If you can't change it, don't worry about it, just stay out of its way.
Do not try to please other people, all that really matters is that you please yourself.
Don't care about what other people think, care what you think.
Don't do what I tell you to do, do what you believe is the right thing to do for you.
Don't do anything you don't want to do unless there is no reasonable alternative. It is doing what you like doing that will be most fruitful.
In words he taught us to be selfish. He spoiled us. He let us do almost anything we wanted to do. He sometimes expressed what he thought the negative consequences might be, but he did not try to control us. All my friends wished they had my parents. It was not called love, but not controlling us was the expression of his love. He told us we were free to face the consequences of anything we wanted to try, but was there to pick us up when we fell down. There could be no greater love.
By his example, in deeds he exuded selflessness, deep respect for the feelings and wishes of others, and encorgagement of others to be all they can be. He taught us how to love. Many have expressed eternal graditude toward dad for encoraging them to change careers, or otherwize fulfill their dreams rather than compromising who they could be and accepting less from life than what they really wanted.
When a group of NJIT colleagues and I incorporated our company, Xanthus, we used my dad's attourney. who asked us if we thought we could conduct business with the same degree of integrity as my dad. After a pause, I said confidently, YES. Hw lowered his eyes and said softly, "I don't think so."
My sons could not come here today but are with us in spirit. Jeff said it well, this should not be about death, but about life, and that dad lived and died as we all should. Their mom put it best, that he has always supported us in so many ways, and most significantly, he taught us how to think
In some ways he had a contrary nature, though he respected others beliefs, he often expressed opposing viewpont in a Socratic fashion. Much interaction with dad was argumentative, Jeff said he argued with him fruitfully for almost 35 years, for me, it is nearly 60 years.
In his final days, a new sweetness of character, formally reserved for my mother, emanated from him. He complimented the health aids and nurces profusely. With happy eyes he would welcome me with, you are a good boy jimmy, I love you. The last days he spoke, I asked how he was doing today, and with great warmth he replied, wonderful.
Although many remember him as a very spiritual person, who influenced their beliefs profoundly, at the same time, he believed in nothing himself. He did not deny the possibility of anything, and was always searching for answers. But he held virtually no doctrine, save the wisdom of Socrates, which he posted prominently on the refrigerator, "The only true wisdom is the knowledge you know nothing.". He challenged the objectivity of any belief in God. In the end, however, he found a notion of God he was comfortable with, the power of truth in a complex world that is beyond any possibility of human comprehensioon.
In his last days he asked my daugher if he was a bad man. She said he was good. He asked what made a person good. She said "the capacity to love". He asked what about the capacity to live?
And today we are here to celibrate the wonderful life he led, and take on the responsibity of continuing his legacy of thingkng out of the box, and living his mandate that we live our lives our own way as he did it his way, wonderfully so.
Mom said it best. "He was a wonderful husband, father and friend. He was exceptional in every way."
My son Jeff said, "everyone should live as he did." He taught us how to think and act, and delegated his thinking and acting to us. In the end I think he believed his legacy would be fullfilled though us. I wish I was so confident. I cant imagine anyone filling his shoes. Together we can only try.
His goodness depends on our capacity to do as he did. For him love was action, not words or feeling, and his love was expressed in how he lived. His religion is not one of belief, it is one of right action being true to oneself, while enabling others to be true to themselves. For me this is not different from loving God, and loving one another as he loved us. Like Jesus, grampscarver is a great example who's time of realization is yet to come as we are only beginning to to learn how to emulate him.
His example demonstrates that to be selfless, we must first be selfish, or we have no self to share with others. His life is a great gift for all of us, and gives us a hope that in remembering him, and following his example, we might live and die well, as he did.
JimScarver
Wednesday, March 25, 2009
Monetary liberation - the cloud monetary system
I am in the process of envisioning an internet cloud market exchange system, bartering essential value and central power currencies, independently based on personal trust and intrinsic value.
Ron Paul is correct in stating centralized money amounts to price fixing, and is contrary to a free market. The taboo against against price fixing is well founded in the literature and history. Monopoly stymies the free market by restricting evolution of the market. In evolutionary game theory, Pavlov, or the learning agent, defeats the bully in the evolutionary game. But this assumes a level playing field. The playing field is not level when one group controls the money supply and thus has a blank check to dominate all other players.
I do not agree with Ron's isolationism and see a gold standard as counterproductive, going from the frying pan into the fire, as the money powers already control the gold market. But nobody is perfect and only one republican and one democrat in the US congress support monetary reform thus far. That we must put the power of money in the hands of the people is the essential common thread.
http://www.youtube.com/watch?v=-r_-QRKyu6g
There is a pyramid of power, dominated by money, and thus the the gamblers with the most potential money. Essential value, belongs to production, rather than gambling. It is imperative, that money largely represents essential value, in a fair contract, to have real value. The international currencies have no essential value and ought not be the primary means of exchange without essential monetary reform. I promote minimalistic monetary reform to eliminart public debt and money as debt in http://MonetaryReformAct.org
reveals many of the essential facts I have yet to clearly express on http://MonetaryReformAct.org but the whole picture is still incomplete.
See also:
But I am an idiot savant, socially inept, lexdysic. It seems I can't know how to do exchange others trust though I am competent in my work and deliver information systems that work. So I leave the challenge to others. I will indeed develop my own exchange for myself, by leave it to others to subjectively trust whatever they trust. Barter may be facilitated according to trust. The system can work immediately as many people mistakingly trust national currencies. This gives alternative currencies the opportunity to gain greater trust.
The infrastructure necessasary, which has been my quest since 1974, now exists on the web.
Re: eies3/web3 etc. http://tech.groups.yahoo.com/group/InfoPhysics/message/6544
The EIES principle of supporting the individuals privacy and power, enabling the individual to facilitate a collective intelligence, is no longer a pipe dream.
Money is a form of trust. The way to facilitate money is to facilitate trust. Not some abstract notion of trust, but trust as it actually exists independently on the cloud. Essential value is due only to trust. Each of us ought to be able to use only money we trust represents essential value. The actual future value is always uncertain but we are not truly free if we are not free to use that which we trust.
I favor decentralized money and free enterprise. I am not partial to capitalism except in the sense of decentralized global value creation. I am not a pure capitalist in spirit and lean toward free and decentralized communal living. In my view, money should not be necessary, except where it is a necessary evil. Free enterprise in a fair market simply maximize our human possibilities.
By "fair market" I only mean not giving control of the money supply to the owners of the private central banks or other elite group. There is no free market if one group has carta blanca, a fair market is a free market. Capitalism is bound to collapse unless we have a free market without centralized control.
I do think we have a responsibility in our collective action to use the power of our pocketbook to encourage market diversity where resources are squandered, monopolies rule or where markets are being displaced. We can thus drive the system toward sustainable growth with minimal discontinuities and suffering.
Gambling is distinct from investing as investing potentially creates essential value, while gambling only seeks to take from others without returning value.
In the present crises no essential value has been lost, instead the money supply created by gambling debt has dissolved making money scarce and causing a collapse. We need a stable currency immune to the activities of gamblers.
I hope to readily assimilate with those of like mind in any respect. What could be better than dreaming and becoming, thus promoting into existence, that which which might be, together?
"History records that the money changers have used every form of abuse, intrigue, deceit, and violent means possible to maintain their control over governments by controlling money and its issuance." -James Madison
The pyramid of power includes central bank owners at the top, multinational corporations, the military-industrial complex, government political machinery, religions, media owners, mafias, and finally the people. Each is has power as allowed by the powers above it, who ultimately rule our destiny and largely determine our success in changing our lives and our world at any level.
"The issue which has swept down the Centuries, and must be fought - sooner or later -is the people versus the banks...Power tends to corrupt, and absolute power corrupt absolutely..." , Lord Acton
Whatever other action we might take will be futile unless we each refuse to use money controlled by some elite. To that end we should use alternate currencies or vouchers, alternate financing, and promote monetary reform.
"Bankers own the earth. Take it away from them, but leave them the power to create money and control credit, and with a flick of a pen they will create enough to buy it back." - Sir Josiah Stamp, former Director, Bank of England
At issue is how we can be effective personally in making alternatives practical.
Jim
http://MonetaryReformAct.org
Jim
http://InformationPhysics.com
http://MonetaryReformAct.org
Ron Paul is correct in stating centralized money amounts to price fixing, and is contrary to a free market. The taboo against against price fixing is well founded in the literature and history. Monopoly stymies the free market by restricting evolution of the market. In evolutionary game theory, Pavlov, or the learning agent, defeats the bully in the evolutionary game. But this assumes a level playing field. The playing field is not level when one group controls the money supply and thus has a blank check to dominate all other players.
YouTube - Ron Paul "The Federal Reserve Is the Culprit!
http://www.youtube.com/watch?v=MpQlB3f9Al4I do not agree with Ron's isolationism and see a gold standard as counterproductive, going from the frying pan into the fire, as the money powers already control the gold market. But nobody is perfect and only one republican and one democrat in the US congress support monetary reform thus far. That we must put the power of money in the hands of the people is the essential common thread.
YouTube - Dennis Kucinich States His Intention To Put The Federal Reserve Under Government Control
http://www.youtube.com/watch?v=-r_-QRKyu6g
There is a pyramid of power, dominated by money, and thus the the gamblers with the most potential money. Essential value, belongs to production, rather than gambling. It is imperative, that money largely represents essential value, in a fair contract, to have real value. The international currencies have no essential value and ought not be the primary means of exchange without essential monetary reform. I promote minimalistic monetary reform to eliminart public debt and money as debt in http://MonetaryReformAct.org
Steve Keen’s DebtWatch No 31 February 2009: “The Roving Cavaliers of Credit”
http://www.debtdeflation.com/blogs/2009/01/31/therovingcavaliersofcredit/reveals many of the essential facts I have yet to clearly express on http://MonetaryReformAct.org but the whole picture is still incomplete.
See also:
URL: http://www.rollingstone.com/politics/story/26793903/the_big_takeover/print
The Big Takeover, The global economic crisis isn't about money - it's about power. How Wall Street insiders are using the bailout to stage a revolution, MATT TAIBBI, Posted Mar 19, 2009 12:49 PM
Grass roots money requires a trusted exchange mechanism. When I was laid off as an information technologist doing brokerage services for Banc of America in the merger with Merril Lynch, the last independent major investment firm not previously owned by an international bank, my friends and their anonymous cohorts at http://WeAreGreat.org suggested I ought do my own stock exchange. A Barclays prime brokerage and BofA I have been in the belly of the beast. I have the necessary experience to do my own exchange. But the idea is ridiculous, but also haunting. I have 25 years of research experience in collaborative systems as well.But I am an idiot savant, socially inept, lexdysic. It seems I can't know how to do exchange others trust though I am competent in my work and deliver information systems that work. So I leave the challenge to others. I will indeed develop my own exchange for myself, by leave it to others to subjectively trust whatever they trust. Barter may be facilitated according to trust. The system can work immediately as many people mistakingly trust national currencies. This gives alternative currencies the opportunity to gain greater trust.
The infrastructure necessasary, which has been my quest since 1974, now exists on the web.
Re: eies3/web3 etc. http://tech.groups.yahoo.com/group/InfoPhysics/message/6544
The EIES principle of supporting the individuals privacy and power, enabling the individual to facilitate a collective intelligence, is no longer a pipe dream.
Money is a form of trust. The way to facilitate money is to facilitate trust. Not some abstract notion of trust, but trust as it actually exists independently on the cloud. Essential value is due only to trust. Each of us ought to be able to use only money we trust represents essential value. The actual future value is always uncertain but we are not truly free if we are not free to use that which we trust.
I favor decentralized money and free enterprise. I am not partial to capitalism except in the sense of decentralized global value creation. I am not a pure capitalist in spirit and lean toward free and decentralized communal living. In my view, money should not be necessary, except where it is a necessary evil. Free enterprise in a fair market simply maximize our human possibilities.
By "fair market" I only mean not giving control of the money supply to the owners of the private central banks or other elite group. There is no free market if one group has carta blanca, a fair market is a free market. Capitalism is bound to collapse unless we have a free market without centralized control.
I do think we have a responsibility in our collective action to use the power of our pocketbook to encourage market diversity where resources are squandered, monopolies rule or where markets are being displaced. We can thus drive the system toward sustainable growth with minimal discontinuities and suffering.
Gambling is distinct from investing as investing potentially creates essential value, while gambling only seeks to take from others without returning value.
In the present crises no essential value has been lost, instead the money supply created by gambling debt has dissolved making money scarce and causing a collapse. We need a stable currency immune to the activities of gamblers.
I hope to readily assimilate with those of like mind in any respect. What could be better than dreaming and becoming, thus promoting into existence, that which which might be, together?
"History records that the money changers have used every form of abuse, intrigue, deceit, and violent means possible to maintain their control over governments by controlling money and its issuance." -James Madison
The pyramid of power includes central bank owners at the top, multinational corporations, the military-industrial complex, government political machinery, religions, media owners, mafias, and finally the people. Each is has power as allowed by the powers above it, who ultimately rule our destiny and largely determine our success in changing our lives and our world at any level.
"The issue which has swept down the Centuries, and must be fought - sooner or later -is the people versus the banks...Power tends to corrupt, and absolute power corrupt absolutely..." , Lord Acton
Whatever other action we might take will be futile unless we each refuse to use money controlled by some elite. To that end we should use alternate currencies or vouchers, alternate financing, and promote monetary reform.
"Bankers own the earth. Take it away from them, but leave them the power to create money and control credit, and with a flick of a pen they will create enough to buy it back." - Sir Josiah Stamp, former Director, Bank of England
At issue is how we can be effective personally in making alternatives practical.
Jim
http://MonetaryReformAct.org
Jim
http://InformationPhysics.com
http://MonetaryReformAct.org
Tuesday, March 24, 2009
Troubleshooting excessive linux iowait time
I hate when a unix system gets really slow but is doing almost nothing. Typically what happens is the load sky rockets, idle time goes to zero, and cpu system and user time usage is very low.
The vmstat command is great for monitoring the situation.
vmstat 30
The vmstat output above shows a system spending most of its time in iowaits with a little more than half of the activity being swapping shared memory.
The machine has 1GB ram and up to 1.5GB swapped in above output. Swapping might be eliminated by adding another gig of memory to the machine. This might alleviate the problem.
You can check which processes are using the most memory with the ps command.
ps aux|sort -n +3|tail -10
apache 27288 0.6 2.5 198840 25684 ? D 08:28 0:03 /usr/sbin/httpd
apache 27923 1.7 4.0 197796 41512 ? S 08:34 0:02 /usr/sbin/httpd
apache 27874 1.1 5.3 214640 54096 ? S 08:33 0:02 /usr/sbin/httpd
apache 27849 0.8 5.6 214648 57504 ? S 08:33 0:01 /usr/sbin/httpd
apache 27852 1.3 5.6 214892 57712 ? S 08:33 0:02 /usr/sbin/httpd
apache 27887 4.1 6.2 273356 63160 ? D 08:33 0:09 /usr/sbin/httpd
apache 27927 7.1 10.9 273356 111236 ? D 08:34 0:09 /usr/sbin/httpd
apache 27890 3.3 11.4 273608 116184 ? S 08:33 0:07 /usr/sbin/httpd
apache 27929 4.0 13.0 290040 133308 ? R 08:34 0:05 /usr/sbin/httpd
apache 27334 2.2 16.7 328072 170808 ? D 08:28 0:11 /usr/sbin/httpd
The above command sorts processes by percent of memory usage (4th col) and shows the ten biggest memory pigs. In this case, the apache httpd processes listed account for about 70% of available memory.
If you still experience high iowait time when the system is not swapping heavily you need to investigate which processes are doing the io. The following scrip puts together vmstat, block_dump, and ps, to show system activity and the largest io consumers during that activity.
bin/whowait
/etc/init.d/syslog stop
echo 1 > /proc/sys/vm/block_dump
uptime
vmstat 5 6
uptime
dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head
echo 0 > /proc/sys/vm/block_dump
/etc/init.d/syslog start
echo Top memory hogs:
ps aux|sort -n +3|tail
Outputting:
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
09:41:06 up 5 days, 20:48, 1 user, load average: 3.90, 3.58, 2.72
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 2 815844 201800 2052 72560 632 831 1141 1802 64 42 18 5 45 32
0 1 815196 277960 2424 72784 1221 0 1298 396 452 624 6 5 1 89
0 0 815196 275816 2560 74004 3788 0 3931 5306 589 753 16 10 6 67
0 0 815196 275816 2560 74004 177 0 442 114 304 175 5 1 72 22
0 0 815196 275716 2564 74004 19 0 19 77 272 98 0 0 98 1
0 0 815196 275344 2608 74136 18 0 25 33 267 114 0 0 96 4
0 0 815196 275344 2616 74136 32 0 32 7 259 93 0 0 99 0
09:41:32 up 5 days, 20:48, 1 user, load average: 2.74, 3.33, 2.66
1074 httpd
782 named
567 kjournald
177 pdflush
130 qmail
130 mysqld
45 java
9 miniserv
6 executable
4 bash
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Top memory hogs:
apache 31245 1.9 2.7 185932 28100 ? S 09:22 0:21 /usr/sbin/httpd
apache 32073 0.4 3.1 190620 32364 ? S 09:36 0:01 /usr/sbin/httpd
apache 32106 1.3 3.1 190608 32176 ? S 09:37 0:03 /usr/sbin/httpd
apache 30959 2.4 3.2 190644 32816 ? S 09:20 0:31 /usr/sbin/httpd
apache 31225 3.0 3.2 191024 33144 ? S 09:22 0:34 /usr/sbin/httpd
apache 30955 1.8 4.0 198840 40964 ? S 09:20 0:24 /usr/sbin/httpd
apache 31982 4.0 5.1 210604 52704 ? S 09:35 0:13 /usr/sbin/httpd
apache 31983 3.1 5.3 212300 54220 ? S 09:35 0:10 /usr/sbin/httpd
apache 31338 1.7 10.8 269196 110512 ? S 09:23 0:19 /usr/sbin/httpd
apache 31924 1.9 10.9 269176 111220 ? S 09:32 0:10 /usr/sbin/httpd
Here the load is not too high and a sample during higher load would be more meaningful.
We may also want to see exactly what files are opened by blocked processes as follows.
for i in `ps -eo pid,user,wchan=WIDE-WCHAN-COLUMN -o s,cmd | awk '$4 ~ /D/ {print $1}'`; do echo "----------"$i"----------"; lsof -p $i; done|more
In the above system the issue to be resolved turn out to be the longevity and memory use of apache processes. This can be achieved examining the apache logs. If you want to see what is accessed recently and have many separate log files to check, the following command is useful substituting the wildcard path to all the access log or error logs on your server.
tail -1 `ls -tr /home/httpd/vhosts/*/statistics/logs/access_log|tail -5`
==> /home/httpd/vhosts/ippsa.org/statistics/logs/access_log <== 68.254.73.3 - - [24/Mar/2009:13:07:57 -0400] "GET /html/index.php?name=News&catid=&topic=12 HTTP/1.1" 200 11618 "http://www.ippsa.org/html/modules.php?op=modload&name=Topics&file=index" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
==> /home/httpd/vhosts/eies.org/statistics/logs/access_log <== 74.247.231.3 - - [24/Mar/2009:13:10:21 -0400] "GET /cgi-bin/ip/herrmann.eies.org HTTP/1.1" 200 184 "-" "curl/7.12.1 (i686-redhat-linux-gnu) libcurl/7.12.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6"
==> /home/httpd/vhosts/wikiworld.com/statistics/logs/access_log <== 68.151.117.181 - - [24/Mar/2009:13:18:25 -0400] "GET /wiki/index.php/ClassicalLogic HTTP/1.1" 200 3497 "http://www.google.ca/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-GB%3Aofficial&hs=2X&q=platonic+reasoning&btnG=Search&meta=" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)"
==> /home/httpd/vhosts/waldorfwithoutwalls.com/statistics/logs/access_log <== 24.5.48.111 - - [24/Mar/2009:13:18:26 -0400] "GET /i/h1articles.gif HTTP/1.1" 200 3085 "http://www.waldorfwithoutwalls.com/articles/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
==> /home/httpd/vhosts/whitescarver.com/statistics/logs/access_log <== 64.66.192.62 - - [24/Mar/2009:13:18:23 -0400] "POST /gallery/Family-Photo-Phun/IMG_1486?full=0 HTTP/1.1" 200 708383 "http://www.whitescarver.com/gallery/Family-Photo-Phun/IMG_1486" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
This shows the last entry in the five most recently updated logs and can easily be adjusted to show more or less. After the fact you can examine each log for the time the problem ocurred to see what was accessed.
One problem that needed to be corrected was spiders (web robots) accessing time consuming functions that need not be indexed and should not be spidered. Entries were added to the robots.txt file in the document root of site to exclude most of the robots from these pages.
It may not be clear which log entries are problematical. To find log entries for specific processes the process id must be recorded in the log. You will most likely need to add %P to the LogFormat in your apache configuration, /etc/httpd/conf/httpd.conf etc.
LogFormat "%h %l %u %t \"%r\" %>s %b %P" common
To be continued...
The vmstat command is great for monitoring the situation.
vmstat 30
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
6 33 1131712 9400 5740 103116 2017 5978 3022 5995 657 855 4 6 0 90
0 31 1172848 106668 2664 41288 3792 4598 4496 5061 726 1094 25 8 0 67
1 31 1146468 8984 1088 19060 2559 674 3034 882 559 642 19 9 0 71
1 30 1238520 15116 2008 22084 2362 4689 2517 4707 712 878 3 6 0 91
1 26 1345528 12440 4756 55188 3861 6525 4744 6729 775 1141 5 9 0 87
1 27 1439400 20520 4952 61852 2641 6133 2892 6889 734 914 6 7 0 87
2 22 1505548 291744 988 17412 3973 5662 4295 5684 802 1194 16 9 0 75
The vmstat output above shows a system spending most of its time in iowaits with a little more than half of the activity being swapping shared memory.
The machine has 1GB ram and up to 1.5GB swapped in above output. Swapping might be eliminated by adding another gig of memory to the machine. This might alleviate the problem.
You can check which processes are using the most memory with the ps command.
ps aux|sort -n +3|tail -10
apache 27288 0.6 2.5 198840 25684 ? D 08:28 0:03 /usr/sbin/httpd
apache 27923 1.7 4.0 197796 41512 ? S 08:34 0:02 /usr/sbin/httpd
apache 27874 1.1 5.3 214640 54096 ? S 08:33 0:02 /usr/sbin/httpd
apache 27849 0.8 5.6 214648 57504 ? S 08:33 0:01 /usr/sbin/httpd
apache 27852 1.3 5.6 214892 57712 ? S 08:33 0:02 /usr/sbin/httpd
apache 27887 4.1 6.2 273356 63160 ? D 08:33 0:09 /usr/sbin/httpd
apache 27927 7.1 10.9 273356 111236 ? D 08:34 0:09 /usr/sbin/httpd
apache 27890 3.3 11.4 273608 116184 ? S 08:33 0:07 /usr/sbin/httpd
apache 27929 4.0 13.0 290040 133308 ? R 08:34 0:05 /usr/sbin/httpd
apache 27334 2.2 16.7 328072 170808 ? D 08:28 0:11 /usr/sbin/httpd
The above command sorts processes by percent of memory usage (4th col) and shows the ten biggest memory pigs. In this case, the apache httpd processes listed account for about 70% of available memory.
If you still experience high iowait time when the system is not swapping heavily you need to investigate which processes are doing the io. The following scrip puts together vmstat, block_dump, and ps, to show system activity and the largest io consumers during that activity.
bin/whowait
/etc/init.d/syslog stop
echo 1 > /proc/sys/vm/block_dump
uptime
vmstat 5 6
uptime
dmesg | egrep "READ|WRITE|dirtied" | egrep -o '([a-zA-Z]*)' | sort | uniq -c | sort -rn | head
echo 0 > /proc/sys/vm/block_dump
/etc/init.d/syslog start
echo Top memory hogs:
ps aux|sort -n +3|tail
Outputting:
Shutting down kernel logger: [ OK ]
Shutting down system logger: [ OK ]
09:41:06 up 5 days, 20:48, 1 user, load average: 3.90, 3.58, 2.72
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 2 815844 201800 2052 72560 632 831 1141 1802 64 42 18 5 45 32
0 1 815196 277960 2424 72784 1221 0 1298 396 452 624 6 5 1 89
0 0 815196 275816 2560 74004 3788 0 3931 5306 589 753 16 10 6 67
0 0 815196 275816 2560 74004 177 0 442 114 304 175 5 1 72 22
0 0 815196 275716 2564 74004 19 0 19 77 272 98 0 0 98 1
0 0 815196 275344 2608 74136 18 0 25 33 267 114 0 0 96 4
0 0 815196 275344 2616 74136 32 0 32 7 259 93 0 0 99 0
09:41:32 up 5 days, 20:48, 1 user, load average: 2.74, 3.33, 2.66
1074 httpd
782 named
567 kjournald
177 pdflush
130 qmail
130 mysqld
45 java
9 miniserv
6 executable
4 bash
Starting system logger: [ OK ]
Starting kernel logger: [ OK ]
Top memory hogs:
apache 31245 1.9 2.7 185932 28100 ? S 09:22 0:21 /usr/sbin/httpd
apache 32073 0.4 3.1 190620 32364 ? S 09:36 0:01 /usr/sbin/httpd
apache 32106 1.3 3.1 190608 32176 ? S 09:37 0:03 /usr/sbin/httpd
apache 30959 2.4 3.2 190644 32816 ? S 09:20 0:31 /usr/sbin/httpd
apache 31225 3.0 3.2 191024 33144 ? S 09:22 0:34 /usr/sbin/httpd
apache 30955 1.8 4.0 198840 40964 ? S 09:20 0:24 /usr/sbin/httpd
apache 31982 4.0 5.1 210604 52704 ? S 09:35 0:13 /usr/sbin/httpd
apache 31983 3.1 5.3 212300 54220 ? S 09:35 0:10 /usr/sbin/httpd
apache 31338 1.7 10.8 269196 110512 ? S 09:23 0:19 /usr/sbin/httpd
apache 31924 1.9 10.9 269176 111220 ? S 09:32 0:10 /usr/sbin/httpd
Here the load is not too high and a sample during higher load would be more meaningful.
We may also want to see exactly what files are opened by blocked processes as follows.
for i in `ps -eo pid,user,wchan=WIDE-WCHAN-COLUMN -o s,cmd | awk '$4 ~ /D/ {print $1}'`; do echo "----------"$i"----------"; lsof -p $i; done|more
In the above system the issue to be resolved turn out to be the longevity and memory use of apache processes. This can be achieved examining the apache logs. If you want to see what is accessed recently and have many separate log files to check, the following command is useful substituting the wildcard path to all the access log or error logs on your server.
tail -1 `ls -tr /home/httpd/vhosts/*/statistics/logs/access_log|tail -5`
==> /home/httpd/vhosts/ippsa.org/statistics/logs/access_log <== 68.254.73.3 - - [24/Mar/2009:13:07:57 -0400] "GET /html/index.php?name=News&catid=&topic=12 HTTP/1.1" 200 11618 "http://www.ippsa.org/html/modules.php?op=modload&name=Topics&file=index" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
==> /home/httpd/vhosts/eies.org/statistics/logs/access_log <== 74.247.231.3 - - [24/Mar/2009:13:10:21 -0400] "GET /cgi-bin/ip/herrmann.eies.org HTTP/1.1" 200 184 "-" "curl/7.12.1 (i686-redhat-linux-gnu) libcurl/7.12.1 OpenSSL/0.9.7a zlib/1.2.1.2 libidn/0.5.6"
==> /home/httpd/vhosts/wikiworld.com/statistics/logs/access_log <== 68.151.117.181 - - [24/Mar/2009:13:18:25 -0400] "GET /wiki/index.php/ClassicalLogic HTTP/1.1" 200 3497 "http://www.google.ca/search?hl=en&client=firefox-a&rls=org.mozilla%3Aen-GB%3Aofficial&hs=2X&q=platonic+reasoning&btnG=Search&meta=" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7 (.NET CLR 3.5.30729)"
==> /home/httpd/vhosts/waldorfwithoutwalls.com/statistics/logs/access_log <== 24.5.48.111 - - [24/Mar/2009:13:18:26 -0400] "GET /i/h1articles.gif HTTP/1.1" 200 3085 "http://www.waldorfwithoutwalls.com/articles/" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7"
==> /home/httpd/vhosts/whitescarver.com/statistics/logs/access_log <== 64.66.192.62 - - [24/Mar/2009:13:18:23 -0400] "POST /gallery/Family-Photo-Phun/IMG_1486?full=0 HTTP/1.1" 200 708383 "http://www.whitescarver.com/gallery/Family-Photo-Phun/IMG_1486" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
This shows the last entry in the five most recently updated logs and can easily be adjusted to show more or less. After the fact you can examine each log for the time the problem ocurred to see what was accessed.
One problem that needed to be corrected was spiders (web robots) accessing time consuming functions that need not be indexed and should not be spidered. Entries were added to the robots.txt file in the document root of site to exclude most of the robots from these pages.
It may not be clear which log entries are problematical. To find log entries for specific processes the process id must be recorded in the log. You will most likely need to add %P to the LogFormat in your apache configuration, /etc/httpd/conf/httpd.conf etc.
LogFormat "%h %l %u %t \"%r\" %>s %b %P" common
To be continued...
Subscribe to:
Posts (Atom)