Engee documentation

Network and Streaming

Julia provides an interface with extensive capabilities for working with objects that perform streaming data input and output, such as terminals, pipes, and TCP sockets. These objects allow you to send and receive data in streaming mode, which means that data is processed sequentially as it arrives. Although this interface is asynchronous at the system level, it looks synchronous to the programmer. This is achieved through the intensive use of Julia’s collaborative multithreading features (coroutines).

Basic streaming I/O

At least the following methods are available for all streams in Julia read and write, which take the stream name as the first argument, for example:

julia> write(stdout, "Hello World");  # подавляет возврат значения 11 за счет «;»
Hello World
julia> read(stdin, Char)

'\n': ASCII/Unicode U+000a (category Cc: Other, control)

Please note that the command write returns 11 — the number of bytes (in the phrase Hello World) output to the stream 'stdout`, however, the return of this value is suppressed using ;.

After that, the Enter key is pressed again so that the Julia environment reads the transition to a new line. As can be seen from the example, write takes the data being written as the second argument, and read — the type of data being read.

For example, to read a simple byte array, you can do the following.

julia> x = zeros(UInt8, 4)
4-element Array{UInt8,1}:
 0x00
 0x00
 0x00
 0x00

julia> read!(stdin, x)
abcd
4-element Array{UInt8,1}:
 0x61
 0x62
 0x63
 0x64

However, this is a rather cumbersome implementation, so there are several more convenient methods. For example, the procedure above can be written as follows.

julia> read(stdin, 4)
abcd
4-element Array{UInt8,1}:
 0x61
 0x62
 0x63
 0x64

If we want to count the entire string, we need the following.

julia> readline(stdin)
abcd
"abcd"

Please note that depending on the terminal settings, you may be buffering lines in the TTY ("terminal teletype"), so you may need to press Enter again to send the 'stdin` data to Julia. When running Julia from the command line in TTY, the default output is sent to the console, and the standard input is read from the keyboard.

To read each line from 'stdin` can be used eachline.

for line in eachline(stdin)
    print("Found $line")
end

And to read individual characters, you can use read.

while !eof(stdin)
    x = read(stdin, Char)
    println("Found: $x")
end

Text input/output

Please note that the above method write works with binary data streams. In particular, the values are not converted to any canonical text representation, but are output as is.:

julia> write(stdout, 0x61);  # подавляет возврат значения 1 за счет «;»
a

Here is the function write outputs a in the stream 'stdout`, and its return value is 1' (since the size of `0x61 is one byte).

For text input/output, use, as appropriate, the method print or show (for a detailed description of the difference between them, see in their documentation).

julia> print(stdout, 0x61)
97

For more information about the implementation of mapping methods for custom types, see Customizable structural printout of the program code.

Contextual properties of the output

Sometimes it is useful to pass contextual data to display methods for output. An object is used as a platform for communicating arbitrary metadata with an I/O object. IOContext. For example, :compact => true adds a parameter to the I/O object with a hint that the display method being called should output abbreviated data (if applicable). In the documentation for 'IOContext` provides a list of frequently used properties.

Working with files

You can write the contents to a file using the write(filename::String, content) method:

julia> write("hello.txt", "Hello, World!")
13

('13` is the number of bytes to write.)

The contents of the file can be read using the read(filename::String) or read(filename::String, String) method as a string:

julia> read("hello.txt", String)
"Hello, World!"

Optional: file streaming

Using the above read and write methods, you can read and write the contents of files. As with many other environments, Julia also has a feature open, which accepts the file name and returns the object 'IOStream`, which can be used to read and write data in a file. Let’s say we have a file hello.txt ` containing the string `Hello, World!.

julia> f = open("hello.txt")
IOStream(<file hello.txt>)

julia> readlines(f)
1-element Array{String,1}:
 "Hello, World!"

To write something to a file, you can open it with the write flag ("w").

julia> f = open("hello.txt","w")
IOStream(<file hello.txt>)

julia> write(f,"Hello again.")
12

If at this stage we look at what is contained in `hello.txt You will see that the file is empty because nothing has been written to the disk yet. In order for the recorded data to be saved on disk, it is necessary to close the `IOStream'.

julia> close(f)

If you review it again `hello.txt `, you will see that its contents have changed.

Opening a file, interacting with its contents, and then closing it is a very common pattern of work. To simplify it, there is another call option. open, which takes two arguments: the function and the file name. It opens the file, calls a function with the file as an argument, and then the file closes again. For example, if there is such a function:

function read_and_capitalize(f::IOStream)
    return uppercase(read(f, String))
end

You can make the following call.

julia> open(read_and_capitalize, "hello.txt")
"HELLO AGAIN."

The file will be opened hello.txt `, `read_and_capitalize will be called for it, and then `hello.txt The file will be closed and its contents, written in capital letters, will be returned.

You can not even define a named function, but use the do syntax, which creates an anonymous function at runtime.

julia> open("hello.txt") do f
           uppercase(read(f, String))
       end
"HELLO AGAIN."

If you want to redirect stdout to a file:

# Open file for writing
out_file = open("output.txt", "w")

# Redirect stdout to file
redirect_stdout(out_file) do
    # Your code here
    println("This output goes to `out_file` via the `stdout` variable.")
end

# Close file
close(out_file)

Redirecting stdout to a file can help save and analyze program output, automate processes, and ensure compliance with regulatory requirements.

A simple example of using TCP

Let’s go straight to a simple example of working with TCP sockets. This functionality is included in the Sockets package from the standard library. First, let’s create a simple server.

julia> using Sockets

julia> errormonitor(@async begin
           server = listen(2000)
           while true
               sock = accept(server)
               println("Hello World\n")
           end
       end)
Task (runnable) @0x00007fd31dc11ae0

The names of the methods will be familiar to those who have worked with the Unix Sockets API, although using these methods is somewhat simpler. The first challenge listen creates a server waiting for incoming connections on the port specified in this case (2000). The same function can be used to create other different types of servers.

julia> listen(2000) # Прослушивает localhost:2000 (IPv4)
Sockets.TCPServer(active)

julia> listen(ip"127.0.0.1",2000) # Эквивалентен первому
Sockets.TCPServer(active)

julia> listen(ip"::1",2000) # Прослушивает localhost:2000 (IPv6)
Sockets.TCPServer(active)

julia> listen(IPv4(0),2001) # Прослушивает порт 2001 на всех интерфейсах IPv4
Sockets.TCPServer(active)

julia> listen(IPv6(0),2001) # Прослушивает порт 2001 на всех интерфейсах IPv6
Sockets.TCPServer(active)

julia> listen("testsocket") # Прослушивает сокет домена UNIX
Sockets.PipeServer(active)

julia> listen("\\\\.\\pipe\\testsocket") # Прослушивает именованный канал Windows
Sockets.PipeServer(active)

Note that the last call has a different return type. This is due to the fact that the server does not listen via TCP, but via a named pipe (Windows) or a UNIX domain socket. Also note that the named Windows pipe has a special format in which the name prefix (\\.\pipe\) contains a unique identifier. https://docs.microsoft.com/windows/desktop/ipc/pipe-names [file type]. The difference between TCP and named pipes and sockets in the UNIX domain is not obvious and is related to the methods accept and connect. Method accept gets a connection to the client from the created server, whereas the function connect connects to the server using the specified method. Function connect accepts the same arguments as listen, so if the environment (i.e. node, cwd, etc.) is the same, then you can pass connect the same arguments as for listening to establish a connection. Let’s try to do this (after creating the server above).

julia> connect(2000)
TCPSocket(open, 0 bytes waiting)

julia> Hello World

As expected, Hello World is displayed. Let’s analyze what’s going on behind the scenes. When calling connect we are connecting to the newly created server. The accept function returns the connection to the newly created socket from the server side and outputs Hello World, indicating that the connection has been established.

An important advantage of Julia is that the API is provided synchronously, although the input and output actually happen asynchronously, so we don’t have to worry about callbacks or even check the server startup. When calling connect the current task waits for the connection to be established and only then continues execution. During this suspension, the server task resumes (because a connection request is now available). It accepts the connection, outputs a message, and waits for the next client. Reading and writing work in a similar way. Let’s take this as an example of a simple echo server.

julia> errormonitor(@async begin
           server = listen(2001)
           while true
               sock = accept(server)
               @async while isopen(sock)
                   write(sock, readline(sock, keep=true))
               end
           end
       end)
Task (runnable) @0x00007fd31dc12e60

julia> clientside = connect(2001)
TCPSocket(RawFD(28) open, 0 bytes waiting)

julia> errormonitor(@async while isopen(clientside)
           write(stdout, readline(clientside, keep=true))
       end)
Task (runnable) @0x00007fd31dc11870

julia> println(clientside,"Hello World from the Echo Server")
Hello World from the Echo Server

As for other streams, use close to disconnect the socket.

julia> close(clientside)

IP Address Resolution

One of the methods is connect, which does not follow the methods listen, is a connect(host::String,port) that tries to connect to the node specified by the host parameter on the port specified by the port parameter. It allows you to do things like the following.

julia> connect("google.com", 80)
TCPSocket(RawFD(30) open, 0 bytes waiting)

The basis of this functionality is the method getaddrinfo, which performs the necessary address resolution.

julia> getaddrinfo("google.com")
ip"74.125.226.225"

Asynchronous I/O

All I/O operations available via Base.read and Base.write, can be executed asynchronously using coroutines. You can create a coroutine for reading or writing data in a stream using a macro @async.

julia> task = @async open("foo.txt", "w") do io
           write(io, "Hello, World!")
       end;

julia> wait(task)

julia> readlines("foo.txt")
1-element Array{String,1}:
 "Hello, World!"

There are often situations when it is necessary to perform many asynchronous operations simultaneously and wait until all of them are completed. To block a program before exiting all the coroutines wrapped in it, you can use the macro @sync.

julia> using Sockets

julia> @sync for hostname in ("google.com", "github.com", "julialang.org")
           @async begin
               conn = connect(hostname, 80)
               write(conn, "GET / HTTP/1.1\r\nHost:$(hostname)\r\n\r\n")
               readline(conn, keep=true)
               println("Finished connection to $(hostname)")
           end
       end
Finished connection to google.com
Finished connection to julialang.org
Finished connection to github.com

Multicasting

Julia supports https://datatracker.ietf.org/doc/html/rfc1112 [multicast] IPv4 and IPv6 using the transport protocol https://datatracker.ietf.org/doc/html/rfc768 [UDP].

Unlike the protocol https://datatracker.ietf.org/doc/html/rfc793 [TCP], UDP makes almost no assumptions about the needs of the application. The TCP protocol provides flow control (speeding up and slowing down transmission to maximize throughput), reliability (automatic retransmission of lost or corrupted packets), consistency (ordering packets by the operating system before sending them to the application), segment sizing, and session configuration and disconnection. There are no such functions in the UDP protocol.

The UDP protocol is commonly used in multicast applications. TCP is a stateful protocol for communication strictly between two devices. In UDP, you can use special multicast addresses for simultaneous communication between multiple devices.

Receiving multicast IP packets

To transmit data using UDP multicast, simply use recv for the socket; the first packet received will be returned. However, please note that this will not necessarily be the first package sent!

using Sockets
group = ip"228.5.6.7"
socket = Sockets.UDPSocket()
bind(socket, ip"0.0.0.0", 6789)
join_multicast_group(socket, group)
println(String(recv(socket)))
leave_multicast_group(socket, group)
close(socket)

Sending Multicast IP packets

To transmit data using UDP multicast, simply use send for the socket. Please note that you do not need to attach the sender to the multicast group.

using Sockets
group = ip"228.5.6.7"
socket = Sockets.UDPSocket()
send(socket, group, 6789, "Hello over IPv4")
close(socket)

An example for IPv6

The following example shows the same functionality as in the previous program, but IPv6 is used as the network layer protocol.

Recipient:

using Sockets
group = Sockets.IPv6("ff05::5:6:7")
socket = Sockets.UDPSocket()
bind(socket, Sockets.IPv6("::"), 6789)
join_multicast_group(socket, group)
println(String(recv(socket)))
leave_multicast_group(socket, group)
close(socket)

Sender:

using Sockets
group = Sockets.IPv6("ff05::5:6:7")
socket = Sockets.UDPSocket()
send(socket, group, 6789, "Hello over IPv6")
close(socket)