I wanted to ask you guys what you think about package managers and external libraries in modern languages, like Python and R.
I learned C in the early 90s and primarily use that for work. It has its standard libraries, sure, but beyond that, I tend to write my own code to do what I want.
Now I'm trying to get up to date with some modern languages (specifically for me: Python and R). I get kind of bummed out following tutorials online. Because I'll be reading a tutorial for something basic like, say, how to parse HTML, and the tutorials start out with "use pip to install Beautiful Soup." But I'm not interested in just learning some tool that some other guy wrote, I want to learn how to actually use python to accomplish these things myself.
Am I looking at this the wrong way? Is the modern way of doing things to download a bunch of external libraries and glue them together? There's something to be said for not re-inventing the wheel, but it's just so different than the way I'm used to doing things. Using external libraries makes me feel like I don't know everything that's going on in my program, but maybe that's just the way things are now?
I'm not trying to start an argument here, I'm just interested to hear what you guys things about this kind of thing.
So how do you track down bugs and optimize your code if you didn't actually write any of it?
David Gutierrez
>Am I looking at this the wrong way? I would say so. Yes, there are downsides to using a bunch of libraries, and I guess just very high level languages like Python. As you mention, less total understanding, less control, less efficiency, etc.
But the upside is that working in Python tends to be insanely more productive than any other language I've ever worked with.
Also take your example >I'll be reading a tutorial for something basic like, say, how to parse HTML Parsing HTML is not basic. Doing this on your own and doing it well without making tons of fuckups and mistakes along the way in insanely hard and time consuming. There is no reason to waste your time on it. And in 99% of cases, you don't usually actually need to concern yourself about how to parse HTML, you just need a certain set of tags from a page or whatever. Let people who like parsing HTML and who work on the library every day and who are likely waaaay better at it than you worry about parsing the HTML.
Camden Nguyen
Hmmmm. Sounds like maybe I should stick to C and assembly.
tldr; OpenBSD replaced sqlite dependency in mandoc with 1000 lines of code, because they didn't need to do everything SQLite does. The result was faster, lighter, and most importantly, easier to maintain.
When maintenance costs ~80% on your average software project, anything that can reduce maintenance is a boon.
Gabriel Butler
From the PDF:
>Three errors: >1. Requirements were not evaluated in detail, but only summarily → selection of excessively powerful, large, heavyweight tools. >2. Integration costs were not evaluated → the tool may save time and code lines for the functionality itself, but may require wasting effort on glue code. >3. Maintenance costs were not evaluated — and it turned out that maintenance was impossible, see the previous slide → that was the fatal error which forced us to redo the work.
Yes! This is exactly what I'm talking about. Thank you, glad to know there are other people with this viewpoint, even if it's in the minority these days.
Easton Adams
yw
This is how I know you're a solid Sr. Dev. The juniors don't understand maintenance costs. Unfortunately, neither does mgmt, usually.
Andrew Sanchez
You guys are fucking delusional. >we used the wrong tool for this insanely specialized scenario >fuck resuability lol ofc you'll note that he did all this work for fucking free.
Eli Bailey
>So how do you track down bugs and optimize your code if you didn't actually write any of it? And what the fuck field do you work in where you have the luxury of only working on code you personally wrote? Because you are in the vast vast minority of developers.
Andrew Lewis
>And what the fuck field do you work in where you have the luxury of only working on code you personally wrote?
I'm the only developer in my department. I work for a major movie studio and write custom tools for production and editorial.
Thomas Sullivan
Id say the ideal would be to strike a balance between not using a million third party libraries and not reinventing the wheel everytime you need a feature. You shouldnt be importing a large library for one function that you use once in your code, thats silly. But say for example im dealing with a situation where i need to be super accurate with my program's math- i cant have any floating point rounding errors or anything like that. Well im kinda shit at making my math accurate to that degree, but i know that Dr. user PhD is really good at it. If he has a library of functions i can use for accurate math, i'd rather use that than risk inaccuracy if i write my own functions.
Of course, the beauty of open source is that if there is an issue i have that stems from me using Dr. user's library, i can let him know so that he can fix it, and its even possible for me to help fix it assuming the fix is within my skillset. So even though issues may when using other peoples libraries, if you are using FOSS (which of course you should be) then using other peoples libraries will be a significantly smaller risk.
Dylan Gonzalez
It was rhetorical, forgive me. Some day someone else it probably going to have to deal with all your code, and unless you created your position you surely must have had to deal with someone else's. Besides, where do you draw the line? Everything you are using is dependent on code you didn't write and do understand. Did you write libc? Gee, how do you debug and optimize code you did not write? Did you write your compiler? Your OS? Your BIOS? Working at an unnecessarily low level of abstraction provides no intrinsic benefit except for perhaps processing performance, at the expense of your own (presumably) valuable time and productivity. Processing power is cheap. Developer time is not.
Leo Moore
behold: the average IT configuration monkey. observe it fumble around as it is incapable of abstract thought and reading between the lines
Jeremiah Rodriguez
>when your philosophy is so absurd that even the most casual and obvious criticism requires ad hominem attacks against someone you literally know nothing about kek
Jason Bell
>not posting the complete version When Jow Forums will learn?
>vague comments unrelated to the discussion at hand are now considered "criticism"
come back with a well-thought-out argument about your views on the openbsd example and i'll take you on, kiddo
but fuck off until you can string two coherent thoughts together without pissing yourself, and your earlier post doesn't inspire confidence.
Robert Adams
>Besides, where do you draw the line? Yes, that's certainly a good point. Even in C, I'm allocating memory with malloc(), not always dropping down to asm to mov and db in memory.
The openbsd example is irrelevant, there is no argument that needs to be made, thanks though.
Brayden Lopez
OP here, just for the record, I'm not this cranky pleb:
Samuel Miller
That was clear, you seem reasonable for the most part. I'm out, good luck OP.
Hunter Nelson
whoopsmissclick
Grayson Campbell
this is about about what i expected, to be honest
>no argument that needs to be made >"insanely specialized scenario" >"he did all this work for fucking free" (with a heavy implication that this is a negative thing)
no, there's plenty to expand into a legitimate argument so that we can hold a proper discussion. but apparently the mental equivalent of banging two rocks together is too hard for you. nice cop-out attempt, though
Xavier Hill
lol, i wonder if you could get away with writing an ffmpeg suite
Jack Hall
lol the document you yourself linked describes how insanely specialized the scenario is. Are you illiterate, did not read it, or cannot comprehend it? >"he did all this work for fucking free" (with a heavy implication that this is a negative thing) Uhh yeah for the vast majority of working programmers, this is so obviously a bad thing I'm just going to assume you're on the spectrum. You will note one very conspicuous missing bullet point from the 'costs' section of that pdf: manpower.
So yeah, not really an argument that needs to be made on my end. Have a good one.
Dominic Evans
ackshually... ffmpeg was causing us a lot of problems in how it wrote Quicktime containers, so I did write our own Quicktime libraries which are fucking sooolid
Ethan Lopez
thats pretty cool, but im assuming the company keeps this kind of thing locked down and private... i'd love to have quicktime containers fixed for ffmpeg but oh well
Camden Powell
i didn't link that document, is my first post in the thread
the example itself is specialized, because it's a specific example. but the process they followed (replacing a large pre-built system with a smaller hand written one) is very common and can be applied to many different scenarios in many vastly different domains of software development. that's what i'm trying to get you to discuss, you dumb cockgargler.
>Uhh yeah for the vast majority of working programmers, this is so obviously a bad thing I'm just going to assume you're on the spectrum. You will note one very conspicuous missing bullet point from the 'costs' section of that pdf: manpower. this very conveniently ignores that people work on projects they're interested in during their free time, to either improve their skills or gather a solid commit record that is very valuable when applying for actual programming jobs. this is obvious that I'm just going to assume you're on the spectrum since you didn't realize it.
So no, you still need an argument. But you don't have to try making one if you don't want to, it's pretty clear that you wouldn't be able to hold a reasonable discussion anyways.
Lucas Gomez
>devil trips
Check out ffmbc if you haven't, they fixed some problems, but not all
Hunter Nguyen
>I don't know everything that's going on in my program, but maybe that's just the way things are now? You are correct that that is the way things are now unfortunately, but it's a fucking terrible state of affairs, as exhibited by all the bloated, laggy, and buggy software being produced now. Just look at the shitshow that is nodejs+npm to get an idea of how bad modern "developer" culture is.
Young devs try to justify this by ignorantly fearmongering about how "bad" software used to be back in the 90's, but even dealing with some of the buggiest dos programs that crashed all the time was never anywhere near as obnoxious as working with some of the "modern" tools these days.
I suspect many subsequent decades will be spent going back and rewriting a lot of the shit that was developed these past two decades, after companies and developers start to notice the huge technical debt hole they've dug themselves into. For example, I fully expect ruby+rails codebases to go extinct long before COBOL finally dies out.
Adrian Jackson
>I fully expect ruby+rails codebases to go extinct long before COBOL finally dies out
But ruby devs are Rockstars!
Srsly tho, man, your post is depressing and true.
Anthony Wright
I'm 25 and I've been in my first programming job for maybe 2 years now. I was just thinking about this the other day, about what modern development seems to be like. The focus shifted from managing your resources and being mindful of code performance to frameworks, package management, and making things flashy. Nobody, as far as I know, had an interest in making a compiler or making their own programming language. Everyone wanted to build an app or make a website and be the next Zuckerberg. I guess I was fortunate enough to start on qBasic and C (I'm from a part of the country that's a little behind the rest) and it gave me a weird perspective. It seems like my generation's lost a sense of elegance in simplicity, if you know what I mean.
Caleb Miller
The libraries for Python and R are all open source. If there's a bug or a function you want to optimize you're free to tinker with the code.
As someone mentioned I don't understand why you would bother rewriting a HTML parser from scratch. It sounds incredibly unproductive.
Also I'm not sure I get where the problem is. I learned programming with Assembly in the late 90's and we used libraries whenever we could. There weren't many, but if some bit of code (say an implementation of a cryptographic algorithm or some math functions) was already available somewhere, nobody would bother rewriting it.
Rewriting every little thing from scratch doesn't sound like a very good practice. With Python packages, you have thousands of users testing the code and reporting issues for thousands of different use cases you would not have thought of. The end result is something much more robust than what you would have come up with.
Isaiah Campbell
people who would have went to wallstreet 10 years ago are going into startups today, and that startup culture carries over to other companies who are always trying to stay relevant.
i'm sure it will blow over eventually during the next recession as jobs get cut, and the next generation of sociopaths will find another industry to take root in.
Andrew Jenkins
>pip Man.. Pip is sane.. Wait til you deal with node/npm.
Noah Torres
You just looked for the wrong piece of info. >parsing html you either understand what you need to do and look for how to do it, ex how to read a file in python, how to parse top-down etc... Or you don't know for shit and get answered "let the library do it"
Asher Peterson
Let me answer you by relating a bit of my experience around javascript. Learning modern JavaScript is tough if you haven’t been there since the beginning. The ecosystem is growing and changing so rapidly that it’s hard to understand the problems that different tools are trying to solve. I started programming in 1998 but only began to learn JavaScript seriously in 2014. At the time I remember coming across Browserify and staring at its tagline, Browserify lets you require(‘modules’) in the browser by bundling up all of your dependencies.” I pretty much didn’t understand any word in this sentence, and struggled to make sense of how this would be helpful for me as a developer. Later I changed my mind. But then, suddenly, I changed it again. I had come a full 360 degrees. That was during my regular expression email parsing days. Any, the simple answer is yes, no, and maybe. hth
Logan Rodriguez
>R Libraries all open source
That's where you're wrong kiddo.
Thomas Martinez
Why the fuck you would want to build an html parser from scratch?
Jonathan Collins
that's just how it is. not reinventing the wheel is nice, but man, is it unfulfilling and the debug/maintenance cycle is fucking maddening
Hudson Sanchez
>Uni life OP, that image hits me on an emotional level. Too often has that been exactly me.
Angel Campbell
use regex to parse and stay away from people telling you to install 3rd party modules
Dominic Morales
>attending parties when you already know that no one gives you any fuck there maybe you should learn how to reject
Lincoln Jenkins
I somewhat agree, as long as the libraries you are using are open source, you are free to maintain them and debug them as if they were part of code you wrote yourself, so no hidden bugs, etc. However, the problem I see nowadays with modern "developer" culture is not only "don't reinvent the wheel", but rather "don't even bother understanding what the fuck is going on under the hood". I could very well write my own HTML parser if I wanted (I actually did years ago as an exercise), but I just don't see a point in doing it when I can use an already existing, better optimized and more general parser, which I can also debug myself, being open source. It's ok to rely on libraries, but it's wrong to think of them as magical black boxes.
