gabriel

I would back it up a little bit and say that any EDR thing would be capable of observing the source of the functions that a program will run and detect outliers. It's a great program to write, everyone should give it a try! It can also be unexpectedly complicated to get all of the corner cases right and you'll drive yourself mad once you try to think of the ways your detection method can be circumvented.

Go deeper. You're right on the cusp of it since all of your questions are fantastic. Even digging into one of your questions will bring up highly relevant material for understanding how a Linux system functions.

    > The sed command is not explained at all. Even the description is non-sense if you don't know already what they are talking about. What is this "default directory", why do you need to set it, why there and only there, why that command works? Even the command itself is something, which I need to check the manual what the heck it does, because it's not a simple one.

By "default directory" they just mean that the upstream GCC source code has a file with default variables and they are modifying those variables with the sed command to use $prefix/lib and $prefix/usr/lib instead of $prefix/lib64 and $prefix/usr/lib64, e.g. lines that contain "m64=" and replacing "lib64" to be "lib". This is what sed is used for: To make string substitutions based on pattern matching. Think through ways of writing the sed command and testing on your own file to see how it behaves. This will lead you to more tools like diff, grep and awk.

    > > -enable-default-pie and --enable-default-ssp
    > The description is almost unusable. So we don't need it, but it's "cleaner", which in this context means exactly nothing. So what happens if I left out? Nothing? Then why should I care?

Go back and re-read the sections up to this point. Make note that you're in Chapter 5, which is the bootstrapping phase for the toolchain cross-compilers. Then look into the features that are mentioned. You can see descriptions by running the ./configure --help most times or looking up the GCC documentation. Those features are for security purposes and if you put that in perspective of the bootstrap phase they aren't needed if the only purpose of the temporary GCC binaries is to compile the final GCC in a later phase. To your point, perform an experiment and enable those features to see if there really is a difference other than time spent to compile. GCC incrementally adds security features like this and they are a big thing in hardened distributions.

    > > --disable-multilib
    > Okay, it doesn't support "something". I have no idea what is multilib, or why I should care. Basically the next arguments' description tells me that because it wouldn't work the compilation otherwise. And then..

A great feature to look up! Check out https://gcc.gnu.org/install/configure.html and search for the option and you'll find that it has to do with supporting a variety of target calling conventions. I can see how that'd be pretty confusing. It has to do with the underlying hardware support for application binary interfaces that GCC can utilize and it turns out you probably only need to support your native hardware (e.g. x86_64). That is, you're compiling your system from scratch and it'll only run on your native hardware (x86_64) but if you were a C/C++ programmer maybe you'd want to have support for other hardware (64-bit ARM systems are pretty common today as an example). So you can save time/space by disabling the defaults and honestly the defaults it includes are just not all that relevant on most systems today.

    > > --disable-threads, --disable-libatomic, --disable-libgomp, --disable-libquadmath, --disable-libssp, --disable-libvtv, --disable-libstdcxx
    > But why would they fail? I want to understand what's happening here, and I need to blindly trust the manual because they just tell me, that "they won't work, believe us".

Try it and find out. I would expect that they would fail due to reliance on other dependencies that may not have been installed or included in this bootstrapped build. Or maybe be/c those components don't behaved well with the LFS-based bootstrap methodology and ultimately aren't needed to bootstrap. Sure trust the LFSers but also think through a way to test your own assertions of the build process and try it out!

    > > --enable-languages=c,c++
    > Why are these the only languages which we need? What are the other languages?

GCC supports many language front-ends. See https://gcc.gnu.org/frontends.html. Only C/C++ is needed be/c you're bootstrapping only C and C++ based package sources. You can validate this as you build the remaining sections. It's conceivable that if you needed other languages in the bootstrap you could include them.

    > So at the end, descriptions are not really helping to understand what's happening, if you don't know already. The last time when I started LFS (about 10 years ago), that was my main problem. That you already need to know almost everything to understand what's really happening, and why, or reading manuals, or trying to find basically unsearchable information (like why libatomic compiling would fail at this step). So after a while, I started the blind copy-pasting, because I didn't have the patience of literary months, and when I realized that this was pointless, I gave up.

It's a steep learning curve, especially of a bootstrap which by its nature is circular! tbh, that's sort of the utility of LFS, it can take you up to a certain point but there are so many options and pitfalls in building a Linux system (or really any complex piece of software) and the real usefulness is pushing through, picking something and learning about it. Then using what you learned to apply to the unknown. GCC is one of the most important packages too, so there's a lot to unpack in understanding anything about it, but the impact is large.

There's also this: https://0xax.gitbooks.io/linux-insides/

I believe the premise of the book is to expose the reader to an element of style that bridges different programming environments. Thus, "style" here is meant more in the terms of a restrictive technique to allow an exposure of these techniques, not writing in idiomatic, or latest-and-greatest Python.

I'd suggest this book even to experienced engineers to focus on different types or a style of programming that he/she may not be familiar. And when you recognize some of the styles it'll be a re-enforcement technique that you were doing something that is seen over and over again (much like a pattern language). I recall going over styles that are seen more in systems programming (like C). The book will contribute to an understanding of code architecture styles across different languages.

For the new engineer, I'd also highly suggest this book as you may begin to see a variety of ways to solve a problem. And I would not concern myself with idiomatic Pythonic styles as that will come with time.

Lisp in Small Pieces: https://pages.lip6.fr/Christian.Queinnec/WWW/LiSP.html

An excellent and very fun read about compilation, interpretation, and programming semantics.

I don’t think you already have to know LISP, although there isn’t an introduction, but if you know any programming it would be enough to pick up the programs that are used as examples. Source code from the book is available and you will end up programming as you move along and it is a delight to see yourself make a program that can interpret another program.

I think you mean Inline::Perl5: https://github.com/niner/Inline-Perl5. It doesn't parse Perl 5 and run it inside of Perl 6 though, it just runs the Perl 5 interpreter for you from a Perl 6 program and lets you marshall data between the interpreters. It's a legitimate technique.

Thanks! I certainly will. It looks pretty cool. I'm certainly learning it and finding the technical details in the documentation to be quite interesting.

I kind of agree with the assessment.

I've been prototyping with VoltDB be/c it has a pretty interesting model that should be able to achieve a near-linear scale of write operations for tables that are partitioned. After reading the docs on VoltDB [1] is became clear to me that they are putting the design constraints up front and if you can work through those constraints [2] you can achieve some wicked scale. But it's a bit more complex than your typical single host database.

The work that VoltDB makes you deal with up front are a lot like the work that would have to be done for a multi-master setup in PostgreSQL to function correctly. I like how VoltDB puts those problems up front, but I'm having problems seeing VoltDB as a general purpose solution. The old PG database I work on right now I can't see in the VoltDB, but maybe parts of it would fit OK.

I look forward to the tooling in PG to get better and better. It's a great community and I do like the work that 2ndquandrant is doing. I like how they approach the community with their work. I do the BDR work is important to understand so that when you're in a situation that calls for it you can take advantage of it (same with VoltDB).

[1] https://docs.voltdb.com/UsingVoltDB/IntroHowVoltDBWorks.php#... [2] https://docs.voltdb.com/UsingVoltDB/DesignPartition.php#Sche...

A good read for API design is "The Little Manual of API Design": http://www4.in.tum.de/~blanchet/api-design.pdf

It falls into the category of non-web API. It's a good read and I think it carries over to web APIs quite well.

Yep, I think that's a pretty big part of the learning process. Realizing that you had a successful build but some things weren't stable. Naturally this would pose the question: What went wrong? To solve that problem you have to figure out a way to debug the system, make a decision as to how to rectify it, apply a fix, then figure out how to verify.

Now imagine the effort that has to go into making Debian work or what if the issue was related to sporadic kernel crashes. It's a lot of fun if you're into that sort of thing.

My first LFS was around 2001, which I think would be LFS v3.0. I can't begin to express how amazing the project was to my learning and development as a programmer.

When I started I was completely new to Linux and the command line. Everything I do today is rooted in the foundation that I learned by doing a close read of the book. The book requires no coding, just command-line, but soon after my first and second reads of the book I began to automate the build process that took me down the path of understanding how distros are put together.

I've only read the book end-to-end once or twice (I think that's all that is needed), but I've done hundreds of builds. Make no mistake: The book is difficult to get through. It requires trial and error as well as a close read of the book to understand why the build is done in the way that it is, which should be explained thoroughly, but if it isn't I'm sure the editors want to hear about the difficulty you are having. There are also bound to be problems bootstrapping from your <insert_favorite_distro>. That's always been a challenge and it's part of the build methodology that LFS uses to create a stable foundation to build a proper system (which relates to the efforts today regarding bit identical reproducible builds).

I work with about a 600 table 2 TB PostgreSQL database and I started to use DataGrip, when it was in beta, for constructing complex queries. It's quite good for writing larger queries, but also taking auto-generated queries from an ORM and re-formatting them to help with debugging application behavior.

For db-specific operations I still use PostgreSQL's command-line client psql. It's an invaluable tool, but it's really nice to have options.

I also like the ability to pin query results so that I can quickly compare result sets in slightly-modified queries that I'm trying to fix.

Not so great at exploring the schema. Could be that I just didn't learn the tool enough to make that easier.

I'd go for an invite if you hae some still. Email in profile.

I'd take an invite as well if anyone sees this, thx :) edit: Email in profile

I'm reading "Lisp in Small Pieces" by Christian Queinnec, which is a book that incrementally builds a scheme interpreter with increasing functionality. I could get away with using Guile to run the earlier examples, but by Chapter 3 I had to use one of the lisp interpreters that the author suggested in his code repo (https://pages.lip6.fr/Christian.Queinnec/Books/LiSP-2ndEditi...). I ended up settling on SCHEME->C and this presentation reminded me of this project and associated work by Joel Bartlett. Checkout the debian code repo (sudo apt-get install scheme2c scheme2c-doc) and the research report for the project: * https://github.com/barak/scheme2c * http://www.hpl.hp.com/techreports/Compaq-DEC/WRL-89-1.pdf

If this presentation appeals to you at all I highly recommend reading Lisp in Small Pieces. It provides a great balance between implementation and theory that you get something running in every chapter and lots of recommendations as to where to go if you want more information.

I'm about half way through and I look forward to my 2-3 hours every night that I spend on it.

I've read many theory and hands-on OS books and this book (the 1st Ed) is my favorite. But it is a dense book! Quite awesomely dense to be honest. One OS to study will certainly help (it helped me) to fully understand why the theory can be useful. This book especially can help make the connection between theory and practice.

I had the same thought. Harper has lots of interesting posts on his blog: http://existentialtype.wordpress.com/ (not to mention his research).

I was delightfully surprised running into this comment in a footnote :)

Here's the direct link to Harper's comment: http://www.pl-enthusiast.net/2014/08/05/type-safety/#comment...

The way that Harper talks about types, such as, "It is not a matter of opinion whether an untyped language is unityped, it is a matter of fact." gives me a different perspective when I program in a language that is effectively unityped.

I know there are subtle differences in the .net memory model compared to mono. I've fallen victim to this sort of thing before and depending on what you are doing and how aware/disciplined a team is, this could very much be a pain point.

Did you all see at the bottom: "COMPUTERS (most of the good stuff is still in papers, here are a few books)"

Most of the good stuff is still in paper!

The site offers alternative downloads of the video file. You should be able to watch these videos on linux with vlc or one of your favorite video players.

Wisconsin here and also see twitter is down. Stuck in a two hour corporate meeting. #terribletiming

Whoa, didn't know about this. I'll check it out.

I always have the same problems when moving from linux over to a Microsoft shop. You really do have to approach VS differently than you would any other dev environment.

Mono is quite practical. However, C# is one thing, and the .NET Memory Model is quite another.

The memory model is, IMHO, where Microsoft is pushing the limits of the existing C# language.

I believe he explains this switch here: http://tratt.net/laurie/blog/entries/fast_enough_vms_in_fast...

It's a very good read. IIRC, the reasons aren't really rooted in the language that is used. It's more about the tooling provided by PyPy for constructing a compiler.

The email equivalent of cold calling people: https://en.wikipedia.org/wiki/Cold_calling

What language prevents TIMTOWTDI? And don't say Python. I'm not talking about the language motto. Because that's the nature of using these things called programming languages: http://www.paulgraham.com/langdes.html

Python has Pep8. Perl has Perl::Critic. C# has FxCop. Etc.

I've not yet seen any language possess the qualities that prevent it from being unreadable. It's an amazing open question though, isn't it?

I think you hit the point with "code maintenance can become a huge headache due to someone not understanding the idioms used by the person who originally wrote the code". But this isn't a language thing. This is a people thing.

I haven't seen anything other than this message. Whether it was last week, yesterday, or right now.

This alone is awesome.

I've had the opportunity to write a small interpreter on the job before. Trust me: I know it is no small task to bite off something like this! Tip of the hat to you.

I'm currently stuck in the .NET world and one of the things I've been working on lately was to have a program fix certain aspects of a medium-to-large code base, but done via semantic parsing of the code base so that I know I'm typesafe and such.

In C# there are open libraries like NRefactor, of course Mono itself, and in the Microsoft world they are working on their own compiler as a service product named Roslyn. Do you think any of these efforts would even help an effort such as your are doing?

I'm asking not so much for C# stuff, but because I feel a momentum coming up that could enable stuff such as a live coding environment (my code is in execution as soon as I write it), and the idea of the debugger is the same as my production run-time (not sort of the same, but really the same). I wish up and coming languages would tackle this stuff head on today.

This is pretty amazing. I regularly have conversations with other programmers where we talk about needing this sort of tool.

Question: For every language, are you writing a full-blown language parser to get the semantic information you need? Can I hook in new parsers to add language support?