Relation between logistic regression and neural network?

Basically, we can think of logistic regression as a one layer neural network.

It is possible to train logistic regression with second-order methods such as newton method. But this computation is not efficient for larger NNs. So the slight different lies in the possible optimizers.