muckrights-sans-merde
bonum fabula frat
### technicalities-of-automated-remastering.html
other pages:
=> bullshit-of-the-week.html bullshit-of-the-week
*originally posted:* aug 2021
i dont think i got to any technical details when i wrote the previous article:
=> how-automated-remastering-works---dont-let-roy-speak-for-you.html how-automated-remastering-works---dont-let-roy-speak-for-you
and thats alright. i wasnt planning to review the technical aspects in more detail, until roy started conflating my work with other tools. at least so its easier to reference, ill go over the history of what ive done in this regard.
first, you need a base distribution--it can be (doesnt have to be) large and ridiculous. as all of these include the linux kernel, i dont actually recommend any of them. linux was a bad move for gnu, (so was hurd, really) one that seems to have changed too many of its goals and priorities for the worse. i dont deny that for a time it was an extremely tempting, even useful kernel. the road its on, i dont recommend, but all the same gnu/linux uses a number of technologies that differ slightly from other operating systems. this shouldnt be surprising.
maybe theres a way to use the remaster tools i made while still using some of the same 3rd-party requirements. thats not a priority for me at all, as a lot of things that are closer to the linux kernel will naturally also become increasingly linux-foundation-and-ibm-like, and id like to get farther away from those. whether or not its technically possible to stay closer to the original tools i used, its not necessarily something you want to do anyway.
to get into some of the technical aspects, lets start with the original goal i had:
it wasnt to remaster anything. it was to take hundreds of puppy linux distros (including ancient ones) and create a DETAILED REPORT of their contents-- not just a list, thats pretty easy. this would produce an html report of all sorts of data with versions, common tools, default window manager, things like that.
imagine a distrowatch for all the puppy distros over the years. that was the idea.
it was just a demo, a thing i was working on to work on something. it was fun though. the best things ive done probably all started as fun experiments, rather than serious projects.
the first thing it needs is the iso image for the live cd or dvd. i realise there are other types of install images (i use those too) but those are less work sometimes anyway, and for now the focus was iso images.
you get the iso, then it mounts the iso. (later i would make it download the iso for you-- unless it was downloaded already).
now that the iso is mounted, it is still read only. later, it would copy the entire tree from the mounted iso to a folder so it could be worked on before being made into an iso again. for just making an html report, this wasnt necessary.
at first, i left the initrd alone. it is possible to read and modify the initrd, and you need to to change the startup scripts or init system, though most of what i wanted to look at was in the large squashfs file. i mostly focused on the sqaushfs file. the location and name varies from distro to distro, so the program looked for it in a growing number of locations. some distros use an ext3 image (yes, in the iso) and the program would simply be modified slightly (by the developer, rather than the user) for such a distro. not much was expected of the user, the script was intended as automatic-- for now its not even a remaster script though.
there are two ways to get the contents of a squashfs file. one is to mount it like you would mount an iso-- whether this is possible depends on your kernel or other system files, though support is common. however, some versions of puppy are so old that the squashfs isnt mountable, so for those unsquashfs has to be used. this is probably a non issue unless youre remastering very, very old distros, but theres nothing wrong with using unsquashfs as the standard solution. this doesnt mean youre limited to remastering distros with sfs files, it simply means those are covered as well.
note that im not going to explain the actual command line to use here, as ive already spent countless hours coding the commands as part of a program. also, im no longer running a system that includes these commands (some of them are present) so its not even trivial to test them.
but first you mount the iso and then you locate and unsquash the sfs. this is all automated, of course.
once you do this, you have access to everything except the initrd. i was able to read information from the initrd later on, but remastering the initrd is a little less trivial than the rest (at least for tinycore, where i was most interested in doing so) and the point so far is simply to expose the filesystem on the live cd. but i grew bored with the html report project and as i got closer and closer to being able to zip the sfs back up and turn it into a bootable iso, i became more interested in that.
i actually did this for two distros at once, the goal was to see if it was possible to simply automate a mix of puppy and refracta (a live devuan derivative). in other words, it became a way to play with the creation of an iso that had the features of devuan but the init of puppy.
this got farther than i thought it would, but later i would drop puppy and focus on remastering refracta to make it more educational (and more automated) and keep it more light. so it wasnt quite deliberate that i got into remastering distros in the first place.
in all of these remaster projects, it wasnt simply a goal to remaster but also to remove non-free stuff. refracta comes with non-free drivers, my remaster removed them. puppy came with non-free software, my remaster was able to remove it. it automatically added and removed applications, it stripped the size down, it made changes to the configuration and the desktop.
my project departed from a number of remaster tools in various ways depending of course on what tool youre comparing it to.
it was not the first automated remaster project-- sugar actually had a tool to remaster ubuntu automatically, to make it into a sugar-based distribution. i found this out later, and was quite pleased to know someone else had done things this way.
but it was different than that project as well.
ultimately the goal was to have a single (text) file that you could download, edit, and share, and that file would remaster your distro automatically.
not only would it remaster it, but instead of downloading a different iso (waste of bandwidth) every time, it let people share a file that was 50-150kb instead of a binary that was thousands of times larger.
sure, it still needed an "input" distro. but maybe that distro comes out with a new image twice a year, and you are releasing new tweaks to your distro every week or two (as i was doing at times). new versions are only 50-150kb, plaintext.
there really wasnt anything else exactly like this-- most tools were distro family specific, this was not. (refracta isnt either, which is one of the cool things about it). most tools require lots of extra space, or even a chroot-- this did not. and that mattered, as i wanted puppy users to be able to use it and a chroot wasnt working (at the time i was new to it and maybe this was my fault-- but chroot was not needed!)
when i used refracta it had more tools available, but this was all developed for puppy first-- to be used from EITHER puppy or refracta, and run from either system. so it was very flexible and avoided solutions and tools that were more elaborate.
for example, it could update binaries in puppy using debian packages (obviously if they didnt run, i didnt use them). apt didnt work from puppy (getting it working was one of the experiments i was trying to find out if it was possible or what it would take) so to copy a package into the filesystem, dpkg-deb was used-- its a very simple to use single binary, like gz for deb packages. and it would even retrieve dpkg-deb from debian and unpackage that, which is more involved.
im increasingly interested in doing things from source, but this was a script that was designed to remaster from binaries. nothing prevented you from using it to work with source also, but it wasnt required or a priority. most remastering is in fact done with binaries, at least for most people (larger projects may vary from this). but my project did not distribute binaries-- it only retrieved them from official sources like apt-get does in debian or pkg_add does in openbsd.
later on, to remaster things like trisquel to remove systemd as the init and replace it with upstart, a chroot was used with trisquels own package manager. incredibly, (a bit incredibly, perhaps) you did not need to be running trisquel to do this-- it worked from refracta. it probably wouldnt work from puppy, but now i was using my existing scripts to remaster things like trisquel and debian to change the default init of the live dvd.
that was a fun hack, though it was only meant to show people "look, you can do this. automatically, even".
most of the time i spent working on this stuff, i spent months if not years remastering and devoted only a day or two to getting trisquel and debian to remaster. thats how flexible this was-- and if you were serious about remastering those things, you would simply tweak it further.
this was a generic, automated, reprogrammable remastering solution. it was not designed as an "application", it was designed to make it possible for people to maintain, not simply create, new distributions based on SMALL communities-- smaller than devuans for sure. and with far less technical expertise.
but its design was simple. it was not rocket science at all.
and this brings in the way it was created in the first place-- i do not like (i know, because ive done it) creating LARGE projects in bash scripts or other unix-like shell scripts. its not because i dont like shell scripts-- i like them very much. i find it incredibly tedious to work on large projects with them, and i dont think bash is ideal for that (but its possible). certainly not for me.
this was a large project. more to the point, i would have NEVER kept working on this as a pure bash or ksh script, and if i had, fewer people would be able to use it on fewer systems (not every version of puppy even has bash, you know? so this could ADD it but it certainly would not use a bash script to add bash...) and i hadnt used ksh much yet so adding bash to distros that didnt have it was something i wanted.
similarly, a lot of puppy users dont know python. they arent interested in python. and yet python has all the string and list/array handling i need (id really rather not automate remastering more than half a dozen distros using an environment that treats so many parts of a "string" using SO MANY different rules. this is why i prefer ksh to bash as well, its more predictable-- but i didnt use ksh yet and i still like python more).
so i dont want to do this in bash.
also, if i do it in python then most puppy users will have to worry about things like editing indentation-- which they want no part of.
i like python indentation. i often use it even when im not writing python code. but thats beside the point, i didnt want it to be a requirement.
fortunately there is a language that has the string and list capabilites of python but requires neither indentation or braces.
it has fewer than 100 commands, and it translates to a language supported on most distros-- including puppy and refracta and openbsd.
and im its author.
so now things like (simple) multiline strings and predictable arrays and stable strings were available features.
i wrote the entire project in a simple language designed for educating beginners and non-coders.
and because it required neither braces or indentation, anybody who wanted to make edits could do things like:
* just delete a bunch of unwanted lines (this isnt unbreakable, but its far less bother than bash or python for people that dont like them)
* move (or copy) a bunch of code somewhere else, because when you dont have worry about indentation or braces this is easier to do. a lot of the time, at least.
im not saying any of these things are impossible, just that theyre easier this way.
and the project wasnt finished-- code can always be made more readable, more manageable, but while the puppy community edition team spent literally years trying to clean up barrys extensive bash scripts, the shell code i used was typically in what ultimately translated to python strings (but not in the source the distro editor had to worry about).
this project wasnt written in python scripts-- it compiles to python scripts.
by doing things this way, it opened the door for programs that actually remaster the remaster script itself, instead of the iso.
maybe youre a serious developer and you think this is silly, but this tool made it trivial for me to do things that other people would spend much more time trying to make work using existing tools.
this was basically a remix lab for potentially all gnu/linux distros. and all i had to do was describe a change, save the file and run the thing.
it remastered puppy, void, tinycore, devuan, debian, trisquel, refracta and others. as for distros too weird to figure out, i couldnt do anything with slitaz. id need more information (im not really interested in working on that, but i did run slitaz years ago). once i had the script, i could spend a short amount of time (maybe a few hours) getting a new distro to work with it. once i did that, the project was just a text file. i even combined several projects into one that could remaster several distros (it was smaller than having a separate script for every distro, but that was also possible).
once you make your changes to the system, it would use (depending on what was available) xorriso or mkisofs to create a bootable iso. reusing the initrd of whichever distro you wanted. the first version had two initrds, one for puppy and one for refracta. and it let you choose which "base" to boot into from the menu, so you could boot puppy with stuff from refracta included, or refracta with stuff from puppy included.
but later on i focused on remastering single distros. i would still use packages from other distros when i wanted to.
when you downloaded this, every script was a complete "receipt" of every single change made. one file listed EVERY CHANGE, so if you didnt like or trust a change you could often SIMPLY DELETE (or comment) THAT LINE. if this was a goal of a distro, it could become more of a hard rule (for the largest section of code in the script at least).
so unlike most remasters, there was no "how did they make this?" it would SHOW YOU how. even if you didnt know how to code, you could probably understand something like:
```
ifequal dist "debianlive09"
now "cp /this/path/file /that/other/path/file" shell
next
```
and if someone wanted, they could change it so that there was a separate text file that configured all sorts of options.
everyone was free to fork this and take it in whatever direction suited them.
however, i had a bigger idea for it-- and if getting away from gnu/linux wasnt more of a priority, id probably still be working on it. but now i dont use gnu/linux.
and that was to automate removing non-free software from EVERY distro. sure, thats more work than i could do myself. however, by making it so everybody could AUTOMATE remastering, it made it easier for small communities to do big things with LOTS of distros.
and i was more than happy to lend a hand.
rather than try to unify every distro through some crap like systemd, i had an idea to make distros more generic.
how do you make more distros more generic?
well, before duix stabbed the whole movement in the back (i would never use their work today) they had what amounted to a generic packaging system. duix (not pronounced deex, but thats what i call those bastard traitors) had an experimental package manager you could add to VARIOUS families of distros.
but thats just package management. and what i mean is, there was also (sadly, github-- sort of beside the point though) a generic distro installer (refracta should sort of count as another one. i never made an installer, but refracta-installer would work on my derivative. i didnt even configure it) which is used in a number of mainstream distributions.
its a bit bloated for my taste but thats beside the point too-- the trend towards distro components like package managers that work on VARIOUS distros and installers that work on VARIOUS distros, a remastering solution that works on VARIOUS DISTROS, can you imagine the sort of freedom that exists in that direction?
basically there are solutions that if people kept working on that sort of thing, you could "eliminate the distro" not by corralling it with some kind of pseudo-proprietary shackles like systemd, but instead "transcend" the entire distro concept by making it modular to the point where-- the very need for something like "debian" falls away, and if you want to make a new distro you can just:
* pick an installer
* pick a package manager
* pick a remastering OR SOURCE BUILDING solution
* start configuring!
and i know, my fellow simplicity lovers are grimacing and thinking "oh no, i can imagine the rust/qt/dbus monstrosity that makes all this possible"-- nope!
no, we are talking stuff that works from the command line, but stuff you could also just open in a text editor, tweak and then run from the term window.
remember: fully automated. so maybe the extra user friendly version has a separate text config-- or even (yuck) an optional gui application that lets you select "options" (but thats not what we are going for. it simply doesnt preclude other people making things like that).
for most of us, (coders and non-coders alike) we would rather edit a text file, run the program, and come back when it spits out the new iso/live image/whatever.
text gives you more power. but ultimately, we want to make code that is easier for more people to understand and alter.
i already worked on that BEFORE i ever made this thing. this thing, as i said, was just a demo.
so--
relatively universal remastering?
completely different styles of scripts designed to be more userfriendly and easy to learn? (hey, the code still needs some refractoring-- but ITS ALSO EASIER TO REFACTOR because its designed to be).
liberating all distros and giving more people the power to remaster whatever they want as a team? (without having to repeat the same steps over and over?)
transcending the distro itself, through more modularity than distros ever had instead of tron-mcp-style takeover by a large corporation?
i realise im hyping the hell out of this, but THIS IS THE STUFF I WAS WORKING ON, and i was trying to promote the IDEAS more than the software. as a BONUS, i had things like "oh, this script automatically changes the default init of debian live in 20 minutes, and all you have to do is RUN THE THING" as a DEMO.
but it wont impress really skilled developers, because i avoided a lot of complex things that really skilled developers would do.
i kept it high-school-science-project simple. because thats more fun (and im not that kind of developer anyway).
this is what you can do with high-school-science-project level tools. and this is why its BETTER to keep software as a PROJECT rather than a PRODUCT-- the latter will always get hijacked, then bloated, then brittle, and finally abandoned.
this project is more or less abandoned, but MOST OF IT is stuff i can (or even do) still use.
more importantly, you should think in bigger terms about what you want from the future of distros. i even started talking about the "end" of the distro.
what did i propose instead? instead of a distro, every thing (debian for example) would be an application: forkable, fixable, something you can modify.
i spent years explaing this stuff, experimenting, demonstrating simple ideas.
and what does roy fucking do? this guy i spent TWO YEARS trying to help out and contribute to what he was doing? he tells people a day or two ago:
```
Wed 22:50:14 │ 〖schestowitz〗 │ you could write scripts to create them
Wed 22:50:25 │ 〖schestowitz〗 │ this is what figosdev did for spins
Wed 22:50:30 │ 〖schestowitz〗 │ I mean, get debian
Wed 22:50:49 │ 〖schestowitz〗 │ then run a bunch of commands to swap around settings, fetch software etc
Wed 22:50:54 │ 〖schestowitz〗 │ no need for another OSI
Wed 22:51:00 │ 〖schestowitz〗 │ rPath and otherd did this
Wed 22:51:12 │ 〖schestowitz〗 │ you had an online configurator
Wed 22:51:18 │ 〖schestowitz〗 │ you make your own distro
```
this is A LITTLE BIT different, though.
never let roy speak for you. or for anybody. always ask for a direct quote, and a url. dont let him bullshit you and tell you he doesnt know where this website is. the url is on his own fucking website. twice. he can find it.
i was working on something anybody could fork for anything. im pretty sure these other projects had countless dependencies.
and thats cool, a lot of important projects have countless dependencies. but sooner or later some dependency decides to throw in the towel, then people who step up to keep it functional (devuan, trinity, mate) get shit on by the likes of debian (or debian. or debian-- because debian shit on ALL THREE of those projects created to keep people happy) and then these projects that are so cool (im not being sarcastic about that) now are "broken" and youre not supposed to use them anymore. this is what oliva and i had to say about that: https://www.fsfla.org/ikiwiki/blogs/lxo/pub/new-dawn.en.html
> Until the importance of valuing, defending and enjoying the freedoms we have conquered dawns on users, they may mistake developer advice as mandate. This renders them vulnerable to social engineering attacks, and unethical vendors may exploit this weakness to subject users of freedom-respecting software to mistreatment that can only be imposed on users of non-free software.
and roy turns that into:
> Thu 04:23:37 │ ... cancelling every person who dared say something not 100% along his own mindset
"sorry" roy, if i feel it necessary to combat the rewriting of history and the takeover by liars and corporate shills with things like "we should make software more modular so that developers have fewer ways to dictate how we do our computing", but maybe you can just fucking read olivas article instead. and shove it up your arse, you lying hypocritical fuck.
not 24 hours later:
```
Thu 18:02:50 │ 〖schestowitz-TR〗 │ we need to get rid of shithub
Thu 18:02:58 │ 〖schestowitz-TR〗 │ like freenode died github needs to die
Thu 18:03:11 │ 〖schestowitz-TR〗 │ but it's propped up by a cabal of proprietary shills
Thu 18:03:22 │ 〖schestowitz-TR〗 │ they hate communities more than they worry about Microsoft
Thu 18:51:17 │ 〖MinceR〗 │ too bad david revoy is an ibm puppet too (his signature is on the anti-RMS letter)
Thu 18:52:58 │ 〖schestowitz-TR〗 │ I know
```
how the fuck are people supposed to get rid of shithub, or deal with shills, if every time they do you turn into a monopolist and liar and smear or advise against ANY AND ALL serious attempts to change anything? then you go right back to doing what you just said other people arent allowed to do, because you said so?
I KNOW THE ANSWER TO THIS!
youre fake, youre the GILDEROY LOCKHART of tech blogging. most of your "activism" consists of manipulating people into solving your problems for you. like your "self-hosting". the bulk of which (including irc, for example) doesnt happen on equipment you own, or can even touch without a plane ticket and someone elses permission.
its never going to be a real debate until you stop lying, you know. not that its going to stop you from pretending.
=> https://muckrights-sans-merde.neocities.org