I only skimmed the page, but i didn't find any references or stats to back up any claims in the text?
535 karma · joined June 2, 2012
Tags don't work without a search feature - we might tag religiously but others don't and won't.
If you're implementing a search feature then you no longer need tags.
It won't take off. Especially as these days we seem to be moving away from files full stop.
Decompressing a 5MP jpg then applying various filters is too lightweight a task to benefit. I thought that would be more or less a perfect GPU task, not so.
Running on OpenCL, the CPU with vector instructions horses a small GPU performance wise for this problem.
If the students are of a different discipline (not aiming to be CompSci grads) or much younger then i think there are better choices e.g. Python.
In a CompSci academic setting i think learning assembly first (that is first programming language but in parallel with the other classes such as computer architecture etc.) is also a valid approach.
Programming from the Ground Up [1] is a very accessible introduction to x86 ASM and even supposing you don't go any further than the examples in this book, you'll still feel the benefits.
[1] http://ftp.igh.cnrs.fr/pub/nongnu/pgubook/ProgrammingGroundU...
Am i being unreasonable in thinking this use of inflationary language is wrong?
If morally you're on the right side of the argument, which i believe the EFF is, then you must play everything completely straight and by the book - otherwise you just give up your advantage.
What language do we reserve for the really bad things.
This is more than likely a factor of my environment though. I'm frequently using Vim "away from home" so to speak.
Best approach for this i've found so far is to set a $VIMINIT on remote login (i.e. an admittedly clunky combination of SendEnv set for Host * in ~/.ssh/config). Still on the lookout for a better way though.
>> All I'm seeing is a constant stream of arrogance
Perhaps not...
Ahh, "man mkpipe", you're mixing up a separate concept - streaming to a remote host vs streaming via a pipe on the filesystem.
>> standard approach in my corner of the world
Syslog streaming has been a common approach since the ~70s i'd guess? Likely before.
>> yet feel entitled to give ops-advice that is at odds with how the rest of the world operates
Hopefully, you see the irony in this.
Yours clearly bends some unbendable truths of the known universe ;-)
A monitoring system ordinarily samples the logfile at a given point in time, that is an open followed by a close operation on the logfile (lest you run into phantom disk usage issues caused by holding an open file handle to a compressed / rotated logfile).
If between sample 1 and sample 2 the log file is rolled, your monitoring system never sees the data between sample 1 and the end of the previous log.
Possible workarounds include logfile streaming (say via a pipe on the filesystem) but that introduces much larger problems and would in no way be compatible with the app rolling its log files.
A much simpler fix is to roll only on events (bounced the app etc.)
>> A script to identify the file-range for a given time-range via binary search is really easy to write
Contrast with no script required. No bugs, no maintenance time, no different versions in different regions. Just no script. The benefits don't stop there - when you need to query, there's no waiting for a bunch of disk seeks while the script runs against a busy disk controller - you can just get straight on with parsing the results.
>> Easier handling. If you need to offload "50GB" from a given log-host then that's easier to do when you know that will be 50 files.
Moving 50 items is easier than moving 1? I'm unconvinced but still open to any interesting ideas here.
>> You don't risk a scratch disk running full
There is no change to the risk profile, neither approach is better in this regard, unfortunately.
>> e.g. interleaved with stack traces
Just one of the many excellent reasons not to log them in production.
>> you must be well aware how long even the simplest merge-job takes on the slice of a single day?
Yes - it runs at full disk I/O speed with almost no CPU overhead. Remember we are merging sorted data (already in temporal order).
>> which happens frequently
It's clearly a high profile app you're working on (in my experience the people in charge of the pennies refuse to sign off on the cost of disk space for this amount of logging without solid reasoning) i'm surprised it behaves as badly as you say. Concerning, but then it also appears you don't have an ops team either.
>> stitch together context from other files.
I get the impression you're thinking that you should manually stitch these together? On the very rare occassions you need this data, remember both are already in time order, we just instantly merge on the fly.
>> A mere grep on a day's worth of logs takes about 10 minutes for us.
It sounds like you've got it all in one file, i.e. not splitting your audit.log out from your error.log from your trace.log? - techniques which I couldn't do without to be honest (the monitoring system would have to consume every byte logged otherwise, when it really only needs to see a subset of total log output - thus saving CPU time and disk seeks).
>> Any kind of text-processing or "for each line perform lookup in $otherplace" quickly pushes that into the hours.
shakes head
Would you humour me with some context here. What is your background? From your replies i would say you are non-ops staff[1], self-taught[2] and with limited experience[3].
[1] You don't demonstrate any knowledge of basic tools, e.g. you mentioned writing a script for a problem solved by a 35+ year old command found on any unix host
[2] You haven't volunteered anything more advanced than obvious / first order approaches. Approaches that i'd expect any CompSci university grad could suggest off the cuff with no prior experience.
[3] You mention grepping through a single logfile of at least 35Gb, likely much larger.
No, It's primarily in the larger log volume systems we run into this problem. Take a FIX session adapter (very common software in the financial services world), baseline is around 5Gb/hour but its susceptible to fairly extreme fluctuation.
When you roll the logs at say 500Mb, there's the chance our monitoring system misses an error in the tail of the previous log as the logs roll (which could in the worst cases mean loosing a couple of million or even more).
Multiple logfiles for a day are harder to search through when handling the most common use case for log files, which is not debugging but ops staff answering questions on 'what happened with X?'
What benefits does a split logfile provide? The one use case is transferring logs across the network. However most log management systems already split large logs up before transferring to the archive silo.
You're loosing a lot, but gaining nothing with split logs.
>> When logging high volumes
It makes no difference, when the market swings you're going to need an extra 200Gb anyway, whether the logs are in chunks or not. In large volume apps deleting to make space is often not an option - you're logging in high volumes for a reason (often regulatory).
>> fetching an arbitrary header from the right file for a given context can be rather painful.
That's been a solved problem for a very long time. Just as recombining a separate trace log with a main app log based on time stamp is a solved problem. I believe this highlights the bigger problem, you can go to university or a.n. other school and learn to be a developer. Where can you go to learn operations? The net result is people come up with the own solutions each time instead of building on work already done. That work is available but its often passed on by word of mouth. Surprisingly hard even in today's world to find decent blogs on ops work (thankfully there are more around now).
>> When a serious error occurs then as much context as feasible must be provided
Yes this is the extraordinarily rare case where stack traces can be useful. They still don't belong in the main app log though.
Push to a separate trace log and recombine on the fly when you need to view. The benefits are to the ops team and the monitoring system.
Another approach not often mentioned is black box logging. Log to a circular buffer in memory, the first bytes of the buffer are set to a pattern that can be searched for in a core dump, the remainder is used as a circular buffer. It's fast to log to, takes no disk space and provides crucial context in the event of a crash.
Im not sure I'll be able to convince you based on your tone, but speaking from experience, you are making the most common mistakes.
I know what the app does, I know the process flows involved but 'failed to covert X at MyClass:44' doesn't have enough context for me.
Also log levels - most developers over use warning and error levels and under uses debug and trace in my experience. It makes filtering out the noise harder.
Rolling log files - roll on an event (new day, process bounced etc.) not on reaching a 10Mb limit or an equally arbitrary 1k lines limit.
Use a header in the logfile, contain at least the version number of the code running, the user and host it is running on.
Lastly stack traces, if they even have a use case at all in production, should not be in the main logfile.
I've learned enough R to be dangerous. Ggplot2 looks gorgeous without trying. It's my current favourite torch for shining on perf and capacity issues (I dream of one day identifying a perf issues root cause with nothing more than the correlation function!)
>> there are many more people like me
Recently I was wondering how can there be so many tech startups around just now. Nothing under the sun is new so how can they cut a living?
Ive realised this attitude (which there is absolutely nothing wrong with) is the reason. The established players leave gaps for their own reasons and the little guy comes along and seizes the opportunity.
Skipping over cheaply accessible best in class technology, at a philosophical level at least, seems akin to pointing a 12 gauge at your toes.
Trivial example, could i choose to always pull out in front of the robot even if it's not my right of way? (presumably safe in the knowledge the robot will always give way to avoid a collision)
Could that escalate to the robot being treated as a mobile speed trap / felony snitch?
Interesting times.
I've been using 2 factor on google for a year or so now. It's not perfect - especially the ways it can be disabled - but it's better than the status quo for now.
A most insightful post, thank you very much.
What's are your views here?
This is a really interesting question for me. I keep putting off a side project that I'd love to do because it suffers this exact problem.
I want to believe there's another way to handle this situation.
I'd imagine there are others like me.
>> NB: The next point is
Yes but I didn't want to write a counter point to each point. I'd taken the view from the listing of possible problems presented that the basics weren't being considered. It reads as a list of random points in a large space rather than a comprehensive listing.
To give a specific example, the article talks about writable concerns when really we want to be ensuring readable gets the same treatment.
>> even if it's in the web directory, can't leak the data without some other compromise
Or a config error (a botched upgrade disabling mod_php could do the trick, or more obscurely a content negotiation oversight), or a leaked backup (another common one - storing random backups under www root)... There's more than one way to skin a cat as they say.
>> If I can even connect
You're often better to view security as something you provide in layers. I wouldn't expect to be able to hit port 3306 from here, but if you're not expecting me to ever be inside your network, then priviledge escalation is going to be fairly quick for me once i surprise you by getting in.
It's not hard, you won't get breathless! ;-)
By the configuration stage, the mental tax can often be as low as checking off items on a checklist.
It is a whole different skill set though and it's trickier in some ways e.g. a developer can often depend on his compiler to tell him if he's messed up the basics.
You could go-live with a host having most attack surfaces battened down tighter than a crab's harris at 50 fathoms, but because nothing told you about DNS tunnelling... (feel free to insert A.N. other rarely talked about but trivial attack)
>> Are directories only writeable by the web server user?
The web server user shouldn't be able to write anywhere except perhaps an upload directory inside its chroot. At no point should it be serving files from that dir (a common mistake).
With tools like SELinux around you can often help cover yourself from midnight mistakes down the line.
A blisteringly common one not mentioned is database authentication details inside .pl, .py or .php scripts.
>> Is PHP installed on your Python server?
Another common one is GCC installed. Often when building perl / python / ruby / nodejs modules or whatever, these will be built on the prod server in small shops. That often means a full compiler suite laying around.
Lots of companies do this already across many sectors.
EDIT: It's still cool though.
My immediate thought is, how do i get myself on that gravy train :-)
>> Everything i believe is important
In the spirit of completeness, you could expand the details of this paragraph:
>> What we want to do is create a unique token which can be sent in an email
There should also be a Time to Live on the token, and it should only be usable once. On the landing page for the link, the user needs to enter* their email address to avoid someone arriving at the url from a means other than the original email.
I feel like i should give an example there, let's say the attacker figured out your random id generation method, say it's a hash of a timestamp for simplicity, if they generate a few thousand links for around 6pm, they may get lucky).
* They don't actually need to enter it, UX guys would be having minor heart attacks at that suggestion, but they could choose their email address from a small table of plausible looking but made-up addresses, or something to that effect, e.g. Bank of America uses a photograph the user setup beforehand.
There is a view that the US actions are unfair and unjust. The delivery method is not terrorism but the end result is indistinguishable in terms of justness or fairness. For example, torturing a man not found guilty of any crimes.
With that in mind, you yourself would be a co-conspirator of the nature you mention your lack of care for above.
If A.N. Other opposing, aggrieved party was to become as effective as the CIA at culling people, such as yourself or your loved ones (in no way are you or your loved ones doing anything "wrong" although you do sympathise with the American unaccountable "bully" philosophy) i'm certain your view would change overnight.
When it comes to scaling up though, Postgres has in my experience, always just worked. This has been true for years. The documentation is excellent and the community extremely helpful and knowledgable.
With cassandra you choose where the extra reads (or writes) to ensure consistency occur as best suits your use case:
* At write time (write quorum) like a traditional replicated datastore
* At read time (read quorum)
* In the background (read one, with a non-zero chance of read repair)
This is quite nice to be able to bend Cassandra on a use case by use case basis (NB: these are not cluster wide approaches, for different columns / circumstances i can choose different patterns).
>> a process can have more memory than address space
How does that work? The best I can come up with is the author is referring to mmap? In which case the address space is still limited to 4Gb although I concede that with the OS pagefile it would be possible to have more than 4Gb in core at a time via judicious use of mmap.
It'll likely be your graphics card uses a portion of your main RAM for its memory. This is common.