12. System Programming in Python

The goal of this seminar is to practice in using Python modules that manage facilities provided by the operation system.

  1. Using the os and sys modules to run programs from Python scripts.

    • Read documentation on the os.path.realpath(path), os.system(command), sys.executable, and sys.argv methods.

    • Study the Python program below.

      • 15-57_dorun.py: 

           1 #!/usr/bin/env python3
   2 '''
   3 '''
   4 import sys
   5 import os
   6 
   7 Python = sys.executable
   8 f = os.path.realpath(sys.argv[1])
   9 
  10 os.system(f'{Python} {f} {" ".join(sys.argv[2:])}')
      This program launches a program specified as command-line argument and passes it the rest of the arguments. The program to be run can look as follows.

      • 15-58_torun.py 

           1 #!/usr/bin/env python3
   2 '''
   3 '''
   4 import sys
   5 print(":", "-".join(sys.argv[1:]))
        • Run the program with python3 and see the result: 
        python3 15-57_dorun.py 15-58_torun.py qwe ert ert
      • Run the program and pass it an invalid program to run. See the output: 
        python3 15-57_dorun.py nothing_torun.py qwe ert ert

  2. Task 01: Modify the 15-57_dorun.py program so that it checks whether the program to run exists before trying to run it.

    • Use the os.path.exists method to check whether the program exists.

    • If the file does not exist, print the "No filename" message to sys.stderr.

    • Save the modified code to the 16-12_dochkrun.py file.

  3. Task 02: Study documentation on the subprocess module.

    • Rewrite the 16-12_dochkrun.py file to run the program using the subprocess.run function.

    • Save the resulting program to the 16-21_dosubrun.py file.

  4. Using pipes to replace shell pipelines.

    • The following shell command: output=$(dmesg | grep sda) can be implemented in Python as follows: 

         1 import subprocess as proc
   2 p1 = proc.Popen(["dmesg"], stdout=proc.PIPE)
   3 p2 = proc.Popen(["grep", "sda"], stdin=p1.stdout, stdout=proc.PIPE)
   4 p1.stdout.close()  # Allow p1 to receive a SIGPIPE if p2 exits.
   5 output = p2.communicate()[0]
    • Execute shell and Python and see the results.

  5. Task 03: On the basis of the previous example, write a Python program that pipelines two commands specified in the command line. Both commands can have any number of arguments.

    • Save the program to the 16-43_pipecmd.py file.

    • Hint: You need to separate one program with arguments from another program with a special separator character. For example, it can be @.

    • The command line can look like this: 
      andrewt@comp-core-i7-3615qm-0dbf32 ~ $ python3 16-43_pipecmd.py date -u @ hexdump -C
00000000  d0 92 d1 81 20 d0 bc d0  b0 d1 8f 20 33 31 20 32  |.... ...... 31 2|
00000010  31 3a 31 33 3a 30 38 20  55 54 43 20 32 30 32 30  |1:13:08 UTC 2020|
00000020  0a                                                |.|
00000021

  6. Using the multiprocessing module to run processes in Python.

    • The example below runs a process using the Process class.

      • 16-50_multiex.py: 

           1 from multiprocessing import Process
   2 import os
   3 
   4 def info(title):
   5     print(title)
   6     print('module name:', __name__)
   7     print('parent process:', os.getppid())
   8     print('process id:', os.getpid())
   9 
  10 def f(name):
  11     info('function f')
  12     print('hello', name)
  13 
  14 if __name__ == '__main__':
  15     info('main line')
  16     p = Process(target=f, args=('bob',))
  17     p.start()
  18     p.join()

      • Note info() here is just a function printing some system info :)

  7. Task 04: Modify the 16-50_multiex.py program. Pass the f function running in another process the additional argument wait, which specifies a delay in seconds. It must be a random value from the range [1..5]. In the f function, print the wait argument together with name. Then delay execution for the number of seconds specified in wait.

    • Use the time.sleep method to delay execution.

    • Use the random.randrange to generate a random value.

    • Save the resulting program in the 16-58_multiex2.py file.

  8. Exchange objects between processes using pipes.

    • Here is an example: 
         1 from multiprocessing import Process, Pipe
   2 
   3 def f(conn):
   4     conn.send([42, None, 'hello'])
   5     conn.close()
   6 
   7 if __name__ == '__main__':
   8     parent_conn, child_conn = Pipe()
   9     p = Process(target=f, args=(child_conn,))
  10     p.start()
  11     print(parent_conn.recv())   # prints "[42, None, 'hello']"
  12     p.join()

    • Note «Pipe» here looks more like socket. It's bidirectional!

  9. Task 05: Modify the 16-58_multiex2.py program to

    • call f() with the only f(conn) argument

    • send name from parent to child via Pipe

    • send the f"Hello, {name}!" message from the child to the parent process. Print it in the parent process.

    • Save the resulting program to the 17-07_multiexchat.py file.

  10. Task 06: Modify the 16-58_multiex2.py program to run multiple processes and wait for them to stop. Use any value (e. g. current number of the process) instead of name argument.

    • Number of processes must be specified as a command line argument (default value is 5).

    • Save the resulting program to the 17-11_multiexmany.py file.

    • Hint:

      1. First, all the Process()-es, storing them in a list

      2. Next, start() all the processes from the list

      3. An then, join() all the processes

    • See the running processes with ps: 

      andrewt@comp-core-i7-3615qm-0dbf32 ~ $ python3 17-11_multiexmany.py &
[1] 2844
0 4
4 3
1 2
3 2
2 2
andrewt@comp-core-i7-3615qm-0dbf32 ~ $
andrewt@comp-core-i7-3615qm-0dbf32 ~ $ ps -ef | grep python
andrewt     2844    2515  0 01:53 pts/0    00:00:00 python3 17-11_multiexmany.py
andrewt     2845    2844  0 01:53 pts/0    00:00:00 python3 17-11_multiexmany.py
andrewt     2846    2844  0 01:53 pts/0    00:00:00 python3 17-11_multiexmany.py
andrewt     2847    2844  0 01:53 pts/0    00:00:00 python3 17-11_multiexmany.py
andrewt     2848    2844  0 01:53 pts/0    00:00:00 python3 17-11_multiexmany.py
andrewt     2849    2844  0 01:53 pts/0    00:00:00 python3 17-11_multiexmany.py
andrewt     2851    2515  0 01:53 pts/0    00:00:00 grep --color=auto python
andrewt@comp-core-i7-3615qm-0dbf32 ~ $ hello 1 2
hello 2 2
hello 3 2
hello 4 3
hello 0 4
[1]+  Завершён        python3 17-11_multiexmany.py

  11. Using pools for parallelizing tasks. The tasks will be distributed among the specified number of worker processes.

    • Here is an example: 
         1 from multiprocessing import Pool
   2 
   3 def f(x):
   4     return x*x
   5 
   6 if __name__ == '__main__':
   7     with Pool(5) as p:
   8         print(p.map(f, [1, 2, 3]))

  12. Task 07: Modify the 16-58_multiex2.py program to run multiple tasks using a pool.

    • Number of tasks must be specified as a command-line argument (default value is 5). The size of pool is fixed and equals 6.

    • The f() function must return the wait value multiplied by 10.

    • Print the aggregated result returned by the Pool.map method.

    • Save the resulting program to the 17-17_multiexpool.py file.

H/W

Finish all unfinished programs. Create the 12_HighLevelLanguages folder at the sugon server. Put all programs there.

Home tasks

Bonus task

Bonus task (if done) should reside in 13_Bonus subdirectory on sugon server. Put there all the files related to your solution, including the prigram itself and 1-million digits pi file it used.

Never use float numbers if you want to achieve precision!

  1. Preliminaries:

    • Take look on Chudnovsky Pi calculation algorithm.

      • $$ \frac{(640320)^{3/2}}{12\pi}=\frac{426880\sqrt{10005}}{\pi} = \sum_{k=0}^{\infty} \frac{(6k)! (545140134k + 13591409)}{(3k)!(k!)^3 (-262537412640768000)^{k}} $$

    • Learn Python decimal data type.

      • How to control precision with getcontext().prec = precision (see example at the very start of documentation)

    • Donwload 1-million precise $$\pi$$ representation (e. g. form here)

      • Remove all non-numeric characters from it
      • Actually we need no more than 40000 digits for ordinary CPU

    • (still no bonus ☺) Write a python program PiChud.py N that iterates over Chudnovsky formula N times and prints out how many exact digits from real $$\pi$$ it has calculated: 

      • george@inspiron:~/src> python3 PiChud.py 4
57
george@inspiron:~/src> python3 PiChud.py 40
568
george@inspiron:~/src> python3 PiChud.py 400
5674

        • This program calculated 75, 568 and 5674 exact $$\pi$$ digits on 4, 40 and 400 iterations of Chudnovsky formula respectively. Your results and time may differ.

        • Use 40000 digits Decimal precision

        • Make sure you're useng only integers and Decimals and no floats

  2. (bonus) Use miultiprocessing to speed up calculations about N times, when N is the number of cores on your computer. Call the resulting program PiChudP.py

    • N can be actually lesser than number of cores. E. g. on 12-core CPU under Linux: 
      george@inspiron:~/src> time python3 PiChud.py 800
11347
python3 PiChud.py 800  19,94s user 0,01s system 99% cpu 19,952 total
george@inspiron:~/src> time python3 PiChudP.py 800
11347
python3 PiChudP.py 800  24,00s user 0,11s system 1002% cpu 2,406 total

      • Parallel version took more userspace CPU time than single-process (24.00s over 19.94s)

      • Parallel version worked 8 times faster than single-process (2.406s over 19.952s)

  3. (more bonuses). Rewrite the program (call it PiChudPA.py) to work N seconds exactly and then stop calculating. Print digits per CPU core statistic (use os.cpu_count()): 

    george@inspiron:~/src> python3 PiChudPA.py 60
33626 digits, 2802.17 per core

Teaser: this is my solution of bonus task № 2. Note you probably will finish with slightly lager code

HSE/ArchitectureOS/Lab_12_HighLevelLanguages (последним исправлял пользователь FrBrGeorge 2020-10-21 00:16:17)