Skip to main content

S3FS + UpdateDB = Billing Increase

By KraFusion

If you are using s3fs to mount an S3 bucket to your server and have noticed a huge increase in ListBucket and HeadObject calls in your Billing. The calls will increase you AWS bill and depending on how many objects you have in your bucket, it could be significant. 

Day over day I saw "Requests-Tier2" - The number of GET and all other non-Tier1 requests - go from 399 thousand to 9 million. That's a 2155% increase and my bill went from 17 cents per day to 4 dollars per day.

Requests-Tier1 - The number of PUT, COPY, POST, or LIST requests for STANDARD, RRS, and tags - also increased, but not as much. 90 thousand per day to 325 thousand per day.




It took some digging, but I found the culprit by running the following command:

$ lsof | grep S3_DROP_ZONE

which gave the following output

[root@ip-999-99-9-999 ec2-user]# lsof | head -1 && lsof | grep S3_DROP_ZONE

COMMAND    PID  TID           USER   FD      TYPE             DEVICE  SIZE/OFF       NODE NAME
updatedb   4211                 root  cwd       DIR               0,36         0    5078268 /S3_DROP_ZONE/redacted/redacted/redacted/81089
updatedb   4211                 root    8r      DIR               0,36         0          1 /S3_DROP_ZONE
updatedb   4211                 root    9r      DIR               0,36         0         14 /S3_DROP_ZONE/redacted
updatedb   4211                 root   10r      DIR               0,36         0     405726 /S3_DROP_ZONE/redacted/redacted
updatedb   4211                 root   11r      DIR               0,36         0    5078257 /S3_DROP_ZONE/redacted/redacted/redacted
updatedb   4211                 root   12r      DIR               0,36         0    5078268 /S3_DROP_ZONE/redacted/redacted/redacted/81089

Solution: Add your mount point (in my case /S3_DROP_ZONE) to PRUNEPATHS in /etc/updatedb.conf so updatedb does not include this when it scans. Locate and kill the PID associated with the "updatedb" process. Shown above, my PID was 4211. Then you can either let the updatedb process start on it own via cron.daily or you can start it manually.

You can monitor the lsof output on your server to verify that its not scanning you S3 mount again. You can also monitor your billing to ensure that it goes down and you can enable request logging on your S3 bucket if you want, to make sure that the requests coming from the server that has S3 mounted on it, is not sending an unusual amount of requests.


Labels:

Comments

Post a Comment

Popular posts from this blog

Dyson AM09 Fan & Heater H2 Error

By KraFusion
No idea what the actual error is and I couldn't find anything useful on the web, so hopefully this will help someone else. I assumed that the H2 error meant that something was dirty, clogged up or that it was overheating because it was dirty or clogged up because the error only showed up when it was in Heater mode. The heater would run for about 30 seconds, then it would show the error and switch over to the high speed fan , I assume to try to blow out the dust. I proceeded to open it up to give it a deep clean because the Dyson instructions for cleaning this thing are ridiculous and don't help at all. Gently wiping down the outside and vacuuming the intake holes....really Dyson, really?! I used cotton swabs and 91% alcohol to clean everything I could get to, starting at the base (in hindsight, this part may not be necessary at all). Then I got to the top of the device where the actual air comes out of and noticed that there was a lot of build up on the heater coils. I could...
26 comments
Read more

ColdFusion to Node.js Conversion : Setup Part 1

By KraFusion
This multi-part post will be community comment driven, which basically means that in an effort to not spend too much time on details that might not matter to the viewers out there, I am only going to share the details which I think are important at the time of publishing the posts. If people (or bots) comment and want more details, I will add them to the original post. As always, constructive criticism is always welcome. Since I am currently using the CFWheels framework (which is inspired by Ruby on Rails or RoR), I chose to try out the Sails.js framework for Node.js (which is also similar to Rails). 1st thing I did was get Node and Apache working together by adding this to my Apache vhost(VirtualHost) block: ProxyRequests on ProxyPass / http://localhost:8124/ And adding this to my hosts file (/etc/hosts): 127.0.0.1 mynewapp.node www.mynewapp.node This allows me to setup an alias in my host file and reference the URL as you normally would without adding the portnumber o...
Post a Comment
Read more

ColdFusion Invalid Image Format Solution

By
For those who have gotten the following error: "java.awt.color.CMMException: Invalid image format" and tried the solutions posted here with no avail. We are going to use the power of Java's JAI (Java Advanced Imaging) library to tackle this one. <cfscript> //path to image imagePath = "pathToImage"; //create java file object, passing in path to image imageFile = createObject("java","java.io.File").init(imagePath); //create a FileSeekableStream, passing in the image file we created fss = createObject("java","com.sun.media.jai.codec.FileSeekableStream").init(imageFile); //create ParameterBlock object and initialize it (call constructor) pb = createObject("java","java.awt.image.renderable.ParameterBlock").init(); //pass in FileSeekableStream pb.add(fss); //create JAI object that will ultimately do the magic we need JAI = createObject("java","javax.media.jai.JAI"); //...
8 comments
Read more