[SOLVED] "spi" missing urllib2 - after python 2-3 update

You have a problem with Salix? Post here and we'll do what we can to help.
User avatar
mimosa
Salix Warrior
Posts: 3311
Joined: 25. May 2010, 17:02
Contact:

Re: [SOLVED] "spi" missing urllib2 - after python 2-3 updat

Post by mimosa »

Like almost all Salix packages that overlap, the python distributed with Salix is identical to the one found in Slackware. It's not at all clear what you're doing, but it hardly sounds like a recipe for stability.
colebaas
Posts: 22
Joined: 18. Dec 2016, 10:55

Re: [SOLVED] "spi" missing urllib2 - after python 2-3 updat

Post by colebaas »

mimosa wrote:It's not at all clear what you're doing, but it hardly sounds like a recipe for stability.
You are very likely to be right except we have an automated script that "rides" the whole system at 100% [dependency checks, loads, hw stability, etc.] and it has been running for the last ~8 hours, throwing out only 3 errors - one of which is spi.
BTW, this has been the "cleanest" run of this automated script in recent memory.
So, we ourselves, are at a loss as to whether to "reinstall" anything?
[ p.s. I work at Loreal/NASA and it's running on a 8x4 nVidia GPU cluster. ]
User avatar
gapan
Salix Wizard
Posts: 6361
Joined: 6. Jun 2009, 17:40

Re: "spi" missing urllib2 - after python 2-3 update

Post by gapan »

colebaas wrote:Running a 1440x server/workstation cluster (Slackware/Salix; probably the largest of its kind) "may" raise some eyebrows.
:shock: I really need to know more about this.
Image
Image
colebaas
Posts: 22
Joined: 18. Dec 2016, 10:55

Re: [SOLVED] "spi" missing urllib2 - after python 2-3 updat

Post by colebaas »

Ohh, hello gapan! I'd be happy to oblige with whatever I can. - Although be advised I'm mostly the "science and hardware guy" and claim to have very limited programming skills. (Really, shamefully limited!)
I do most of the work in Mathematica, haskell and Erlang so probably wouldn't be able to serve up any major Python "insights", myself. Nevertheless, we have some people around here who stand a really good chance to address just about anything intelligently.
colebaas
Posts: 22
Joined: 18. Dec 2016, 10:55

Re: [SOLVED] "spi" missing urllib2 - after python 2-3 updat

Post by colebaas »

@gapan

While I am unclear about the scope of your interest regarding our setup but while we're waiting on our scripted burn-in load to conclude, I figured drop in some notes here.
It's probably too long so abandon it at will.
---

Now I may have to eat my own words - below - if we really managed to break the system this time around but since it's being continuously abused by our automated script for the past 23+ hours while remaining rock-solid.... ... I'll take my chances and try to give you a general idea about our setup here. (Assuming that is what you were interested in?)


In broad strokes the hardware is standard by current availability. (it's basically a "Ferrari" but without windows, seats, review mirrors, and all the other superfluous fluff.)

There are 2 Clusters, each with 8 nodes.
Each node is configured with
2x Xeon E5-2699 V3; 2.2 GHz | 72 threads
4x Tesla P100 16GB | 4x3584 FP32 cores
2x 750GB PCIe SSD (800k IOPS) | 5GBs / 4GBs
Inifiniband(12x) swithched fabric 100 Gbs (Latency<<0,0001 ms)
Attached is a 360 TB storage cabinet.

The workstations are all essentially
2x Xeon E5-2699 V3; 2.2 GHz
2x GeForce GTX 1080 8GB
etc...

As far as base OS and software;
I like our people to have a wide array of choices
thus, they can select from a list:
Slackware 32 [ past, stable, future, don't care ]
Slackware 64 [ - " - ]
For those who feel the above is too limiting, we also offer: Multilib.
(higher bit-ness and/or precision is not always better - It, however, is always slower! — Insert nVidia Spec sheets here and observe the hard-coded compiler flags!)

On the workstations as far as DEs, one, or all, of the following:
rxvt, tmux or i3.
(For those who don't like it... Have a nice day! One of the key benefits of being an unelected dictator, also known as a department head, I don't have to practice democracy or even observe it. - So I don't.)

Now why would anyone - people often wonder or laugh - run Slack on all this shiny hardware... (with Salix as the fine pkgmgr.)
Well, because... Do you just hope to go fast or do you actually want to get there?
And since at the end of the day Linux is Linux, one might as well start building from a sane, non-stupified (suckless) environment.

For instance... just off the top of my head;
- Does anyone still remember the still recent Debian disaster with SSH (encryption)? If you were in a similar business to ours that would've been a career-ender, guaranteed!
- How about trying to log into a current ubuntu server 16.04 LTS ("Moron Meerkat " or whatever its called) after a fresh install? - Good luck! Its (still) broken.
- Would anyone like to know how many ways does a fresh CUDA install break RHEL 7? - I would tell you if only I could count that high.
- Has anyone actually seen a NetworkManager.conf file since systemd broke out? - Where 64 individual NICs must be (manually!) disabled just so a cluster would boot? (Within the life expectancy currently prevalent on this planet, that is.)
- Arch is the "bleeding edge"! - It's "bleeding" alright, though, yet to notice the "edge"! (It has a great Wiki, though. May be they are really just an electronic publishing company??)
- "Yeah, but I want to boot fast!". OK, our nodes boot under 3 seconds. - Beat that! Now, POST is an entirely different matter. You can boot in 3 seconds after you have waited for an hour and a half for 1400 servers to post. If you are running anything with systemd you can safely leave for an extended vacation... and never even have to return, ever!
- How about CoreOS, the greatest thing since sliced bread? Excellent idea, still. - And by "still" I mean that we have known about cgroups (containers) for about 30 years and have been running them since the 2.6 kernel, which is over a decade now. So if you can't do better than them by now with a vanilla kernel and a little patching, throw in a couple Slackware packages, then are you really sure you should be touching computer things instead of just loading up Mint?!
- A little while ago if one got a job at Google, they all started their first day by "throwing an ubuntu server" onto the company workstation. Ask them, how many were still installing and reinstalling software weeks later, before writing a single line of code for their new employer. (Most of them switched to a Mac since - or get fired.)
- And many more...

From personal experience, for whatever it's worth;
In the past 6 years (2010), starting every fresh install on any new hardware, Slack or Salix has never broken, failed or corrupted a single piece of hardware, firmware, code or package, relevant in our experience. Thank you all!!
At the same time we have gone into space (and left the solar system), fullfilled multi-hundred million dollar contracts - and lost laptops with those contracts on them; but which were fully and reliably encrypted, what is currently impossible with any systemd inclusion -, and all the while we observed our less fortunate although more "hip" compadres doing the dance of the village idiot, trying to fix every stupid, kakameni, still-born idea that Lennart (Poettering) and the Red Hat afficionados have dropped on their unsuspecting laps. (Remember Pulse Audio?Try encrypted root! Want the whole list? How much time do you have?)

What we really do here on a daily bases is destroy as much really expensive hardware as humanly possible. Burn them out faster than anyone around. You know why?... ... because one can.
When your stuff is up and running 100% of the time, under maximum load, 24 / 7 / 365, with a perfect record for the past 6 years, then that's what happens. It also allows occasionally to get things done. (Even important or at least somewhat cool stuff.)
And when you manage to do the above you vaporize an awful lot of silicon in the process, regardless of who made it, how well and how much did it cost. (The newest Pascal cards have a projected life expectancy of about 8-12 months, when used "properly" - 100% load. The first one just went up in smoke here, after 7 months in service.)

All in all, while Slackware is certainly not the end-all, I myself remain hard-pressed to think of a better, more reliable and more economical starting point in all of OSS-land. It is rather easy to make the case for efficiency, measured in decision units, when it comes to Slack's well-curated simplicity while people in general, have a tendency to overestimate their aptitude and operational capacity, none more so than when it comes to complex computing environments.

In our case, as it turns out, once off this planet, one rarely gets to turn around or reboot, or so it seems.
One also seldom requires the most recent driver for some $2 junk silicon, carved with an exacto-knife out of sandstone in a Malay jungle and sold as the sound-card in a shiny new laptop.

But if you ever decided to land on a planet, in a far far away galaxy,.... (many decades or centuries post launch) ... ...without knowing anything about the place in advance (is it liquid, solid, gas, in between??).... ....we may be able to help you out, even if just a little bit.
colebaas
Posts: 22
Joined: 18. Dec 2016, 10:55

Re: [SOLVED] "spi" missing urllib2 - after python 2-3 updat

Post by colebaas »

This time it's really SOLVED!!
Found the problem and It was due to our own stress-testing script and the way it migrates the environments between the workstations and the backup machines.
[ The script re-writes the library paths on the fly but as it turns out "someone" forgot to swap one step, to make sure that actually both Pythons are fully setup before it tries to run spi and company.] It simply tried to run spi under 3 while still migrating 2.
Sorry about the stupid wild-goose chase.
Much obliged for the inputs though!! Cheers

[ EDIT ] System-wide [general] Fix

There are two separate issues effecting system's own Python environment

1] When installing Anaconda3 >= 4.1.1 it will set the Python path environment variable ( PYTHONPATH) and thus, grab the system's original python environment - and likely break most of your packages and stuff.
- Unset this variable and/or delete it from /etc/profile.d/anacondaX.sh and all is well, except now Anaconda wont be able to find its own version of python.
The fix: Probably the easiest is just to set the necessary few bash aliases to initialize Anacondas own Python environment. (One could also softlink python from /usr/bin/python to /usr/local/foo but then make sure to check the $PATH for the order in which python will be found - first come first served! It's easy to break stuff this way!
If you wanted to find the packages which depend on the system's own version of python and then check if everything is properly restored afterwards, you can run something like this

Code: Select all

# egrep -R '#! */usr/bin/env +python' /usr/bin
2] If you have the unfortunate need to install Intel's MLK Libraries and including the Studio Package Stub, that will also helpfully set the PYTHONPATH for you and break your stuff all over again! If you are extra unlucky - latest Intel Studio stub - it will start linking stuff "for you" all over the place. The likely location to look for softlinks [to delete] is /etc/profile.d/... /usr/bin/.... /usr/local/.... One - or all - of these will contain obvious Intel stuff - called Intel_foo_ver_something. (just gather them all up and dump them into their own folder under your home directory or some place convenient.) Then you'll have to recreate their original softlinks - locally, in their new folder, otherwise the binaries will not load because their stupid licensing agent checks for ... who knows/cares what.

These should fix any and all remaining system's Python issues.
spi and everything else "python" is functional again.


[Note] The take away from all of the above is this;
A) It took us quite a while to track down these otherwise glaring issues simply because we were running an automated script in a headless environment and thus, weren't tipped-off by a Graphical DE crumbling before our eyes. spi was the canary in the coal mine.
B) Most of the sources for these libraries have their own python front-ends and about half of them still make two assumptions - usually hard-coded in their Make files;
1) Your system does not have it's own version of Python; Or 2) If it did then it must be somewhere in some package manager's root - such as Anaconda.
Rarely, if ever, does either of these assumptions hold true.
Happy computing!!...
User avatar
gapan
Salix Wizard
Posts: 6361
Joined: 6. Jun 2009, 17:40

Re: [SOLVED] "spi" missing urllib2 - after python 2-3 updat

Post by gapan »

colebaas, thanks a lot for taking the time to write all of this information here! What you're doing there is certainly impressive!

And I'm glad you figured everything out too!
Image
Image
colebaas
Posts: 22
Joined: 18. Dec 2016, 10:55

Re: [SOLVED] "spi" missing urllib2 - after python 2-3 updat

Post by colebaas »

@gapan

You are most welcome. Hopefully it may be useful to some other unsuspecting Python "upgraders".

I would also like to thank you for your tireless effort over the years!
[We have been lurking around since the days of zenwalk but because of the nature of our business and also due to my own, personal aversion to chiming in, rarely do we get to express our appreciation.] Again, many thanks!
Post Reply