Skip to main content

S3FS + UpdateDB = Billing Increase

By KraFusion

If you are using s3fs to mount an S3 bucket to your server and have noticed a huge increase in ListBucket and HeadObject calls in your Billing. The calls will increase you AWS bill and depending on how many objects you have in your bucket, it could be significant. 

Day over day I saw "Requests-Tier2" - The number of GET and all other non-Tier1 requests - go from 399 thousand to 9 million. That's a 2155% increase and my bill went from 17 cents per day to 4 dollars per day.

Requests-Tier1 - The number of PUT, COPY, POST, or LIST requests for STANDARD, RRS, and tags - also increased, but not as much. 90 thousand per day to 325 thousand per day.




It took some digging, but I found the culprit by running the following command:

$ lsof | grep S3_DROP_ZONE

which gave the following output

[root@ip-999-99-9-999 ec2-user]# lsof | head -1 && lsof | grep S3_DROP_ZONE

COMMAND    PID  TID           USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
updatedb   4211                 root  cwd       DIR               0,36         0    5078268 /S3_DROP_ZONE/redacted/redacted/redacted/81089
updatedb   4211                 root    8r      DIR               0,36         0          1 /S3_DROP_ZONE
updatedb   4211                 root    9r      DIR               0,36         0         14 /S3_DROP_ZONE/redacted
updatedb   4211                 root   10r      DIR               0,36         0     405726 /S3_DROP_ZONE/redacted/redacted
updatedb   4211                 root   11r      DIR               0,36         0    5078257 /S3_DROP_ZONE/redacted/redacted/redacted
updatedb   4211                 root   12r      DIR               0,36         0    5078268 /S3_DROP_ZONE/redacted/redacted/redacted/81089

Solution: Add your mount point (in my case /S3_DROP_ZONE) to PRUNEPATHS in /etc/updatedb.conf so updatedb does not include this when it scans. Locate and kill the PID associated with the "updatedb" process. Shown above, my PID was 4211. Then you can either let the updatedb process start on it own via cron.daily or you can start it manually.

You can monitor the lsof output on your server to verify that its not scanning you S3 mount again. You can also monitor your billing to ensure that it goes down and you can enable request logging on your S3 bucket if you want, to make sure that the requests coming from the server that has S3 mounted on it, is not sending an unusual amount of requests.


Labels

Labels:

Comments

Post a Comment

Popular posts from this blog

Dyson AM09 Fan & Heater H2 Error

By KraFusion
No idea what the actual error is and I couldn't find anything useful on the web, so hopefully this will help someone else. I assumed that the H2 error meant that something was dirty, clogged up or that it was overheating because it was dirty or clogged up because the error only showed up when it was in Heater mode. The heater would run for about 30 seconds, then it would show the error and switch over to the high speed fan , I assume to try to blow out the dust. I proceeded to open it up to give it a deep clean because the Dyson instructions for cleaning this thing are ridiculous and don't help at all. Gently wiping down the outside and vacuuming the intake holes....really Dyson, really?! I used cotton swabs and 91% alcohol to clean everything I could get to, starting at the base (in hindsight, this part may not be necessary at all). Then I got to the top of the device where the actual air comes out of and noticed that there was a lot of build up on the heater coils. I could...
26 comments
Read more

ColdFusion Invalid Image Format Solution

By
For those who have gotten the following error: "java.awt.color.CMMException: Invalid image format" and tried the solutions posted here with no avail. We are going to use the power of Java's JAI (Java Advanced Imaging) library to tackle this one. <cfscript> //path to image imagePath = "pathToImage"; //create java file object, passing in path to image imageFile = createObject("java","java.io.File").init(imagePath); //create a FileSeekableStream, passing in the image file we created fss = createObject("java","com.sun.media.jai.codec.FileSeekableStream").init(imageFile); //create ParameterBlock object and initialize it (call constructor) pb = createObject("java","java.awt.image.renderable.ParameterBlock").init(); //pass in FileSeekableStream pb.add(fss); //create JAI object that will ultimately do the magic we need JAI = createObject("java","javax.media.jai.JAI"); //...
8 comments
Read more

Fixing DNS issue on MacBook Pro

By
This may be an issue with 10.6(Snow Leopard) altogether or just with the MacBook Pros, but every once in a while the internal DNS settings get hung up or crap out or just stop working! You're connected to the internet, you've checked the connections, you've got on other devices , you've asked co-workers if they are having issues with the internet, you've pulled your hair out, banged your head against the wall, you get my drift. Whatever you do or try your $2000 MacBook Pro will not do one simple thing CONNECT TO THE INTERNET!! Hopefully this solution will work for some of you and save the hair on your head. Run these commands from terminal(copy command from sudo to plist): sudo launchctl unload /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist THEN sudo launchctl load /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist and bingo bango everything is just peachy again!
8 comments
Read more