Experimenting with Ethereum Smart Contracts As Quickly As Possible

30-Aug-2020

Updated 24-Aug-2021 to reflect changes: epirus from 1.4.0 to 1.4.1 (fixes weird extra "generate" verb)
Updated 8-Apr-2023 to reflect changes: epirus is back to web3j and v1.4.2; geth and solc have later revs AND locations

Smart contracts and Ethereum are all the rage. So how can you write and run a Hello World in solidity, the defacto language that compiles to the Ethereum virtual machine (EVM), and call it from a Java client? This is not about transferring ETH from one wallet to another. That is ... "easy". We're talking about experimenting with smart contract call types, return types, and storage dynamics in solidity and interacting with it in a Java client.

Turns out it is a harder that you think -- mostly because the existing documentation on this topic (and there is a lot of it) starts with you doing some of the following:

Setting up a whole Ethereum node (e.g. geth) which connects to peers and syncs over time and you have to set gas limits and it takes a minute to commit transactions even on the testnet and you have to get the genesis block correct and ...
... or setting up an account on an Ethereum provider like Infura which eliminates the install issues but still has gas and other considerations
Hand-editing solc (the solidity compiler) output
Picking a client side library that supports the JSON-RPC protocol of the node.
Getting anything to compile.
The documentation for web3j contract wrapping is somewhat separated (and rightfully so) from the setup of the ethereum node.
Everything depends on maven which drags in 20 libs you never heard of and probably will clash with jars you already have.

The key here is to use geth and web3j. Here's how -- from scratch. It will take all of 5 minutes. And most important, you will understand why what utils and execs and libraries are being used. This example has no hidden magic; only what you download and run is used. In particular, no maven transitive closure on dependencies is used.

Step 1: Get the Ethereum software

Go to https://github.com/ethereum/solc-bin and download the binary version appropriate for your test environment. If you pull down a Linux build, you will get a tar file with (effectively) a single member: the fully linked executable geth. Let's put that tar in /tmp and extract it into our own opt and symlink it into our own bin:

$ cd $HOME/opt
$ tar xf /tmp/what-you-downloaded.tar 
$ ls -l 
./geth-linux-amd64-1.9.20-979fc968
./geth-linux-amd64-1.9.20-979fc968/COPYING
./geth-linux-amd64-1.9.20-979fc968/geth
$ ln -s ./geth-linux-amd64-1.9.20-979fc968/geth $HOME/bin
$ geth version
Geth
Version: 1.11.5-stable
Git Commit: a38f4108571d1a144dc3cf3faf8990430d109bc4
Git Commit Date: 20230321
Architecture: amd64
Go Version: go1.20.2
Operating System: darwin
GOPATH=
GOROOT=

OK. At this point you know you can run the geth bits.

Step 2: Set up a development Ethereum node using `geth`

On your Unixish platform of choice, create a directory like this:
```
$ cd $HOME
$ mkdir -p eth/dev/data
```
The eth part is not terribly important BUT mindful that you will likely want to move from pure dev to real network engagement at some point, we need to create a dev/data subdirectory. The dev part is especially important. We'll see why later when we want to move to production.
Start the geth server in "dev" mode. The dev subdirectory name is not at all linked to the option --dev; we could have called the subdirectory snowball. Now remember this is the first time we are doing this (from scratch!):
```
$ cd $HOME/eth/dev   # NOT in the data directory; one above it
$  geth --dev --datadir data --http --http.api admin,debug,eth,miner,net,personal,shh,txpool,web3
```
OK, so a lot is going to happen here.
- The --dev option sets up several behaviors that you want for a single node development server. You can look HERE for more information, but the important things are that with --dev:
  - You don't need peers or funding.
  - --dev sets up the server with a dynamically created "developer account" that is already funded. You don't have to transfer ETH. You don't have to hit the rinkeby testnet faucet and slurp a few ETH. --dev magically creates the coinbase and a developer account linked to the coinbase already funded with plenty of ETH. It also set the gas price to 0 so that you can test all your smart contracts over and over and over including really expensive and silly mistakes but it is OK because the operations will still complete AND you can see the gas spent as a result in the TransactionReceipt.
  - It generates a BLANK PASSWORD wallet file in the $HOME/eth/dev/data/keystore directory. More on this soon!
- Many docs say just start the dev server with geth --dev. This is bad for two reasons:
  1. Without the --http option, you cannot connect to the node from your web3j Java client.
    IMPORTANT! This option enables an endpoint at http://127.0.0.1:8545
  2. Without the --http.api options, many of the JSON-RPC functions issued from your web3j Java client will not work.
  3. Without the --datadir option, the resources the geth server creates are either in /tmp or in memory. This includes the wallets and prefunded account! Without the --datadir option, if you boot the machine or just Ctrl-C out of the running of the geth server, when you restart it (again without --datadir), a whole new set of keys, files, etc. are created -- and this will complicate your integration of your Java client!

If later on when deploying a contract or calling a state-changing function you get an error similar to "Error processing transaction request: only replay-protected (EIP-155) transactions allowed over RPC" then add

     --rpc.allow-unprotected-txs

to the geth command line. This clearly would not be recommended for real production but our examples here are all in development/experiment mode.

I recommend running the geth command in the foreground in a separate shell just so you can see the diagnostic output. When you get more comfortable the environment, you can background it easily:

$ cd eth/dev
$ nohup geth --dev --datadir data --http --http.api admin,debug,eth,miner,net,personal,shh,txpool,web3 >> $HOME/geth.log 2>&1 &

Remember: Because you have set up the --datadir, you can kill -15 or otherwise stop the geth server at any time and restart it. After the developer resources are created the first time, they will be reused the next time you start your dev server with the same --datadir option.

AGAIN
The next time you restart `geth` it will NOT recreate the developer resources in `eth/dev/data/keystore`. It will use them again, thus allowing you to NOT change all your Java client connections and keystore resources!

Step 3: Get the web3j a.k.a epirus libs

epirus is the new web3j.
It is not https://www.epirussystems.com.
It is not https://epirus.org.
It is right here:

$ curl -L get.epirus.io | sh

Note the -L option for following redirect. This does NOT require root or privileged access.

After you do this, a set of libs will be installed in $HOME/.epirus and your PATH will be expanded to include $HOME/.epirus. You can tell if the install worked by executing this:

$ epirus
  ______       _                
 |  ____|     (_)               
 | |__   _ __  _ _ __ _   _ ___ 
 |  __| | '_ \| | '__| | | / __|
 | |____| |_) | | |  | |_| \__ \
 |______| .__/|_|_|   \__,_|___/
        | |                     
        |_|                     

epirus [OPTIONS] [COMMAND]

Description:

... more

Step 4: Get the solidity compiler

You can see all your options at https://solidity.readthedocs.io/en/v0.7.0/installing-solidity.html
but for our purposes in the Linux world get this: https://github.com/ethereum/solidity/releases/download/v0.8.0/solc-static-linux
This is a binary compile Linux x86_64 ABI compatible executable. It is not a tar or an rpm; it is simply the executable. Let's put it in $HOME/opt:

$ ls -l solc-static-linux 
-rw-r--r--@ 1 swguy  staff  11346488 Dec 11 16:11 solc-static-linux
$ mkdir -p ~/opt/solc-0.8.0
$ mv solc-static-linux ~/opt/solc-0.8.0
$ ln -s ~/opt/solc-0.8.0/solc-static-linux $HOME/bin/solc
$ ls ~/bin
lrwxrwxrwx 1 swguy staff      55 Aug 30 22:26 geth -> /home/swguy/opt/geth-linux-amd64-1.9.20-979fc968/geth
lrwxrwxrwx 1 swguy staff      55 Aug 30 22:26 solc -> /home/swguy/opt/solc-0.8.0/solc-static-linux
$ solc --version
solc, the solidity compiler commandline interface
Version: 0.8.0+commit.9e61f92b.Linux.g++

OK. At this point you know you can run the solc bits.

Step 5: Prep the solidity and Java software environment

Start by creating some directories:

$ cd $HOME
$ mkdir -p projects/myroot   # our home base
$ cd $HOME/projects/myroot
$ mkdir lib   # This will simplify classpaths later on

$ # Symlink the big web3j jar into our lib
$ ln -s ~/.epirus/epirus-cli-shadow-1.4.0/lib/epirus-cli-1.4.0-all.jar lib  # if the 'epirus' exec worked before then so will this

$ mkdir -p src/main/sol/contracts                 # where the Hello World smart contract goes
$ mkdir -p src/main/java/com/yourcompany/web3/apps    # where the Hello World java caller goes

Step 6: Write a Hello World solidity contract

Here it is as a link: TheContract.sol and shown below.
Make sure this ends up as $HOME/projects/myroot/src/main/sol/contracts/TheContract.sol


pragma solidity ^0.7.0;

// SPDX-License-Identifier: MIT

contract TheContract {

    // Some contract-specific data.
    struct Loan {
	bytes cparty;
        uint32 amount;
    }

    Loan loan;  // "autoconstructed"

    constructor(bytes memory cparty, uint32 amount) {
	loan.cparty = cparty;
	loan.amount = amount;
    }

    /* Things that generate state change cannot return data.  They only return
     * a TransactionReceipt object.  Only view or pure functions will be
     * wrapped with nice type specific return values.  Even if you declare
     * this function to return something, it is IGNORED in the Java wrapper.
     * You will NOT get 
     * the types back in the Java-wrapped function; you have to wait until the
     * state change is committed to the block and then you can ask for the data
     * with a view scoped function.
     */
    function changeCounterparty(string calldata newCpty) public {
	loan.cparty = bytes(newCpty);
    }

    //  Returning a string instead of bytes means in Java you get a String
    //  instead of byte[] which is friendlier.
    function getCounterparty() public view returns (string memory) {
	return string(loan.cparty);
    }

    //  An example of a function that returns two types
    function getInfo() public view returns (bytes memory, uint amount) {
        return (loan.cparty, loan.amount);
    }

    //  structs e.g. Loan are internal to contracts.  You can pass and receive
    //  structs from an internal function, but you cannot do so to the outside
    //  world i.e. there is no Java wrapper for Loan.
}

Step 7: Compile the contract

Do this:

$ cd $HOME/projects/myroot
$ solc --abi --bin --overwrite -o build/main src/main/sol/contracts/TheContract.sol
Compiler run successful. Artifact(s) can be found in directory build/main.

What has happened is that two artifacts have been created:

build/main/TheContract.abi This is the interface spec to our smart contract.
build/main/TheContract.bin This is the actual solidity EVM opcode in hex.

Step 8: Wrap the contract

$ cd $HOME/projects/myroot
$ epirus generate solidity --binFile=build/main/TheContract.bin --abiFile=build/main/TheContract.abi --outputDir=src/main/java --package=com.yourcompany.web3.generated

This will create src/main/java/com/yourcompany/web3/generated/TheContract.java. This is the heart of the integration. This interface shields Java apps from many of the tedious aspects of JSON-RPC integration and does Java-friendly type conversion too. Look at TheContract.java to get a sense of what just happened turning the ABI and BIN files into ... this.

Step 9: Write Your Java Clients

Pay close attention to the wallet/credential inputs! The wallet file is what geth created in --datadir in the first run of --dev mode. Of course your file basename will be different -- but the path ($HOME/eth/dev/data/keystore) should be the same.
We have two programs here: one to deploy the contract, and the other to call functions on it after it has been deployed. Of course you could put this all into one file but especially for experimenting with just the Java functions it is faster to not redeploy an unchanged contract over and over, even on the fast development geth server.
Here they are as links: deploy1.java and access1.java and shown completely below.
Make sure these end up in the target directory as:
$HOME/projects/myroot/src/main/java/com/yourcompany/web3/apps/deploy1.java
$HOME/projects/myroot/src/main/java/com/yourcompany/web3/apps/access1.java

// deploy1.java 

package com.yourcompany.web3.apps;

// Super important: Our app uses the generated Java from solc!
import com.yourcompany.web3.generated.TheContract;

import java.math.BigInteger;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.web3j.crypto.Credentials;
import org.web3j.crypto.WalletUtils;
import org.web3j.protocol.Web3j;
import org.web3j.protocol.http.HttpService;

import org.web3j.tx.Contract;  // for GAS_LIMIT constant
import org.web3j.tx.ManagedTransaction; // GAS_PRICE constant


public class deploy1 {

    private static final Logger log = LoggerFactory.getLogger(deploy1.class);

    public static void main(String[] args) throws Exception {
        new deploy1().run();
    }

    private void run() {

    try {
        // This is the geth dev server you are running.  Note: NOT https!
        String MY_CONN_URL = "http://127.0.0.1:8545";

        // This is the developer wallet autocreated and funded by the --dev option
        // when we started geth.
        // There should be only one file in the keystore directory; use that one.
        // The actual name will be different than this.  Again, the point of using
        // --datadir with the --dev option is that this resource will be created 
        // just once and you don't have to keep finding a new wallet file and 
        // change the program:
        String MY_WALLET_FILE = "yourhome/eth/dev/data/keystore/UTC--2020-08-30T15-35-42.179527700Z--ad2a9bde6c3fe0149c29db01ed40966b1b06f3d0";
        String MY_WALLET_PWD  = ""; // IMPORTANT! The password for the geth --dev developer account wallet is BLANK!

	System.out.println("Connecting to " + MY_CONN_URL);
        Web3j web3j = Web3j.build(new HttpService(MY_CONN_URL));

	System.out.println("Loading credentials...");
        Credentials credentials =
            WalletUtils.loadCredentials(MY_WALLET_PWD, MY_WALLET_FILE);

	System.out.println("Deploying smart contract; be patient...");

        TheContract contract = TheContract.deploy(
            // These first 4 args are always the same types:
            web3j, credentials, ManagedTransaction.GAS_PRICE, Contract.GAS_LIMIT,

            // The next are the required args for the constructor() of the smart contract!
            // OR no additional args if the constructor has no args.
            "CPTY1".getBytes(),  // notice: we pass byte[], not String
            new BigInteger("1000")  // ... and ALL uint types go in (and out) as BigInteger
        ).send();

        String contractAddress = contract.getContractAddress();
        System.out.println("Contract address: " + contractAddress);


        web3j.shutdown();

        } catch(Exception e) {
            System.out.println("exception: " + e);
        }
    }
}

// access1.java 

package com.yourcompany.web3.apps;

// Super important: Our app uses the generated Java from solc!
import com.yourcompany.web3.generated.TheContract;

import java.math.BigInteger;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import org.web3j.crypto.Credentials;
import org.web3j.crypto.WalletUtils;
import org.web3j.protocol.Web3j;
import org.web3j.protocol.core.methods.response.TransactionReceipt;
import org.web3j.protocol.http.HttpService;
import org.web3j.tx.Contract;  // for GAS_LIMIT constant
import org.web3j.tx.ManagedTransaction; // GAS_PRICE constant
import org.web3j.tx.Transfer;
import org.web3j.utils.Convert;
import org.web3j.utils.Numeric;

import org.web3j.tuples.generated.Tuple2;


public class access1 {

    private static final Logger log = LoggerFactory.getLogger(access1.class);

    public static void main(String[] args) throws Exception {
        new access1().run(args);
    }

    private void run(String[] args) {

	String contractAddr = args[0];

	try {
        String MY_CONN_URL = "http://127.0.0.1:8545";

        String MY_WALLET_FILE = "yourhome/eth/dev/data/keystore/UTC--2020-08-30T15-35-42.179527700Z--ad2a9bde6c3fe0149c29db01ed40966b1b06f3d0";
        String MY_WALLET_PWD  = "";

	System.out.println("Connecting to " + MY_CONN_URL);
        Web3j web3j = Web3j.build(new HttpService(MY_CONN_URL));

	System.out.println("Loading credentials...");
        Credentials credentials =
            WalletUtils.loadCredentials(MY_WALLET_PWD, MY_WALLET_FILE);

	TheContract ct2 = TheContract.load(contractAddr, web3j, credentials,
			   ManagedTransaction.GAS_PRICE, Contract.GAS_LIMIT);

	String cpty = ct2.getCounterparty().send();
	System.out.println("cpty: " + cpty);

	// State changing function do NOT return regular types, only a 
	// TransactionReceipt
	TransactionReceipt TXr = ct2.changeCounterparty("FOO").send();
	System.out.println("  TX status:   " + TXr.getStatus());
	System.out.println("  TX root:     " + TXr.getRoot());
	System.out.println("  TX TXhash:   " + TXr.getTransactionHash());
	System.out.println("  TX blk hash: " + TXr.getBlockHash());
	System.out.println("  TX blk num:  " + TXr.getBlockNumber());
	System.out.println("  TX gas used: " + TXr.getGasUsed()); // this is particularly cool
	System.out.println("  TX from:     " + TXr.getFrom());
	System.out.println("  TX to:       " + TXr.getTo());

	// Thanks to uint256, all integers coming out of smart contracts use
	// BigInteger, not int or long.  And you cannot communicate with structs,
	// only basic types.  The org.web3j.tuples.generated package provides
        // up to 20 returns (Tuple2, Tuple3, ... Tuple20)
	Tuple2<byte[],BigInteger> result = ct2.getInfo().send();
	System.out.println("info: " + result.getValue1() + "," + result.getValue2());

        web3j.shutdown();

        } catch(Exception e) {
            System.out.println("exception: " + e);
        }
    }
}

If you feel bold, please go ahead and factor out the common bits -- especially the credentials setup -- into a Utils class that both deploy1.java and access1.java can consume.

Step 10: Compile Your Java Client

You can use whatever build tools you wish but for this experiment we are going low level and using javac directly so you can very clearly see that all we need is our 2 app sources, the generated wrapper class, and the (enormous) epirus a.k.a. web3j jar. Java 8 is the minimum rev required. Java >=9 has slight issue with the http connection hanging up upon exit. The workaround is to call OkHttpClient.connectionPool().evictAll() after shutdown, e.g.

    web3j.shutdown();
    OKHttpClient.connectionPool().evictAll()

and this will allow main() to exit. The httpClient object can be extracted or otherwise made visible in a variety of ways; it is left as an exercise for the reader but a decent approach is to create your own client just as the logic buried in Web3j.build(new HttpService(MY_CONN_URL)) would:

    private static OkHttpClient createOkHttpClient() {
        final OkHttpClient.Builder builder =
            new OkHttpClient.Builder().connectionSpecs(CONNECTION_SPEC_LIST);
        return builder.build();
    }

    public MyWeb3Wrapper(String conn_url) {
        this.httpClient = createOkHttpClient();
        this.web3j = Web3j.build(new HttpService(conn_url, this.httpClient));
    }

    public Web3j getweb3j() {
        return this.web3j;
    }

    public void shutdown() {
        this.web3j.shutdown();
        this.httpClient.connectionPool().evictAll();
    }

Let's compile with Java 8 for now:

javac -d build/main  -cp build/main:lib/epirus-cli-1.4.0-all.jar  src/main/java/com/yourcompany/web3/generated/TheContract.java  src/main/java/com/yourcompany/web3/apps/deploy1.java src/main/java/com/yourcompany/web3/apps/access1.java

Step 11: RUN!

$ java -cp build/main:lib/epirus-cli-1.4.0-all.jar  com/yourcompany/web3/apps/deploy1
Connecting to http://127.0.0.1:8545
Loading credentials...
Deploying smart contract; be patient...
Contract address: 0xe41eaccbe13fc9228bc0f5a33d0114f4fb8ae95c

Now copy that newly minted address and use it for access1:

$ java -cp build/main:lib/epirus-cli-1.4.0-all.jar  com/yourcompany/web3/apps/access1 0xe41eaccbe13fc9228bc0f5a33d0114f4fb8ae95c
Connecting to http://127.0.0.1:8545
Loading credentials...
cpty: CPTY1
  TX status:   0x1
  TX root:     null
  TX TXhash:   0x974c1127e1a4553d3c88e638e63e4128cc088d4d2abeeb3a229d7400686d185d
  TX blk hash: 0xd3ef76487d2dc9f751ce949c861d97d87e296cc04f9320574c8474bfb9406238
  TX blk num:  34
  TX gas used: 28842
  TX from:     0xad2a9bde6c3fe0149c29db01ed40966b1b06f3d0
  TX to:       0xe41eaccbe13fc9228bc0f5a33d0114f4fb8ae95c
info: [B@21aa6d6c,1000

Success! Note how returning bytes through the wrapper emerges as ... byte[] which you can easily turn into a string. Or you can convert to string on the way out of the function as in getCounterparty.

Later on: Moving to Production

The --dev mode makes it easy to experiment but moving to production requires attention to at least these things:

State change operations are much slower in production. You might have to put these changes on a different thread (very likely away from the GUI!)
Wallet and security management is essentially bypassed in --dev mode including using a BLANK password. You will have to adopt much better techniques and protocols for managing these resources.
If you run a real geth make sure you use a different --datadir, e.g. mkdir -p $HOME/eth/prod/data
The epirus a.k.a. web3j jar is ... big, even though 1.4.0 cut the size in half:
```
-rw-rw-r-- 1 swguy staffr 57030250 Nov 12 12:51 epirus-cli-1.4.0-all.jar
$ jar tf epirus-cli-1.4.0-all.jar  | wc -l 
34998

-rw-rw-r-- 1 swguy staff 108540653 Aug 18 17:56 epirus-cli-1.2.4-all.jar
$ jar tvf epirus-cli-1.2.4-all.jar | wc -l
65358
```
To avoid maven versionitis, the web3j folks now release this single mega-bundle jar filled with everything: web3j, bouncycastle, reactive java, okhttp, gradle, google common, etc. etc. etc. This certainly makes our experiments easy to compile and run but in real apps that have other jar dependencies and release controls like asset stamping, you will have to pare this down to something more managable that contains only the web3j bits and the minimal set of libs.

Like this? Dislike this? Let me know