Go-CoNLLU – Some Much Needed Machine Learning Support in Go

data has a better idea - machine learning golang

The post Go-CoNLLU – Some Much Needed Machine Learning Support in Go appeared first on Qvault.

Python is commonly seen as the AI/ML language, but is often a dull blade due to unsafe typing and being slow, like really slow. Many popular natural language processing toolkits only have Python APIs, and we want to see that change. At Nuvi, we use Go for the majority of our data processing tasks because we can write simple and fast code. Today we are open-sourcing a tool that has helped make our ML lives easier in Go. Say hello to go-conllu.

What is CoNLL-U?

The Conference on Natural Language Learning (CoNNL) has created multiple file-formats for storing natural language annotations. CoNLL-U is one such format and is used by the Universal Dependency Project, which hosts many annotations of textual data. In order to use these corpora, we need a parser that makes it simple for developers to utilize the data.

Universal Dependencies Machine Learning Logo
Universal Dependencies

How Does Go-Conllu Help?

Go-conllu parses conllu data. It is a simple and reliable way to import conllu data into your application as Go structs.

The GoDoc can be found here with the specifics

Let’s take a look at the example quick-start code from the Readme. First, download the package.

go get github.com/nuvi/go-conllu

Then in a new project:

package main

import (
        "fmt"
        "log"

        conllu "github.com/nuvi/go-conllu"
)

func main() {
        sentences, err := conllu.ParseFile("path/to/model.conllu")
        if err != nil {
                log.Fatal(err)
        }

        for _, sentence := range sentences {
                for _, token := range sentence.Tokens {
                        fmt.Println(token)
                }
                fmt.Println()
        }
}

All the sentences and tokens in the corpus will be printed to the console.

If you need a .conllu corpus file you can download the Universal Dependencies English training model here: en_ewt-ud-train.conllu

Thanks For Reading

Hit me up on twitter @wagslane if you have any questions or comments.

Follow me on Dev.to: wagslane

The post Go-CoNLLU – Some Much Needed Machine Learning Support in Go appeared first on Qvault.



source https://qvault.io/2020/06/08/go-conllu-some-much-needed-machine-learning-support-in-go/

Comments

Post a Comment

Popular posts from this blog

Why is Exclusive Or (XOR) Important in Cryptography?

Image
If you are getting into cryptography, or just trying to understand the fundamentals, you may have noticed that the exclusive-or operation is used quite often, especially in ciphers. What is XOR ( ⊕ )? XOR, or “exclusive or” operates on binary data. It returns true if both of its inputs are opposites (one false and one true), otherwise, it returns false. An example in go code would be something like: func exclusiveOr(a bool, b bool) bool { return a != b } XOR Cipher – The Perfect Cipher The XOR operation can be used as a simple cipher for encrypting and decrypting messages with a single key. This is known as symmetric encryption. It is interesting to note that if: The key is the same size as the message The key is kept secret and generated truly randomly Then the cipher is impossible to crack. This is known as a one time pad . However, a simple XOR shouldn’t be used in production due to the key length needing to be too long to be practical. Cipher Example A...
Read more

Base64 vs Base58 Encoding

Image
By Lane Wagner –  @wagslane  on Twitter Base64 is one of the most popular encoding formats for representing data. Have some binary data? Base64 encode it for convenient readability and parsing. Base58 is just another encoding format (with 58 characters instead of 64, and has gained popularity largely due to Bitcoin and other cryptocurrencies. When it comes to data encoding, there is typically a trade-off made between: Human Readability: Do humans have a good idea of what is being represented at a glance? Efficient data compression: How many bytes are used to represent the same data? Let’s rank Base58, Base64, and ASCII encoding against each other using these metrics! First: A Note on Binary All data is stored in a raw binary format on computers. These encoding formats (Base64, Base58, and ASCII) are just different ways of reading and writing binary data. For example, in Base64 the binary code 000000 represents the letter A , but in ASCII the binary 00000000 repres...
Read more

(Very) Basic Intro to Hash Functions (SHA-256, MD-5, etc)

Image
This is going to be a basic introduction to hash functions. I will assume most of my audience is here to gain an understanding of why hash functions are used and the basic idea of why they work. My goal is to explain it in a general sense, I will be omitting proofs and implementation details and instead focusing on the high-level principles. Why Use A Hash Function? Hash functions are used all over the internet in order to securely store passwords, find duplicate records, quickly store and retrieve data, and more. For example,  Qvault  uses hashes to extend master passwords into private encryption keys. A longer list of uses here:  https://en.wikipedia.org/wiki/Hash_function#Uses I want to focus on several important features of hash functions, arguably the most important features. Hash functions scramble data deterministically No matter the input, the output of a hash function always has the same size The original data can not be retrieved from the scrambled d...
Read more