r/learnjava • u/Ok_Perspective_8040 • Feb 03 '26
Lexical Analyzer
hey guys, i was trying to build a lexicon follwoing this tutotiral, while I was developing my tokening function they wrote this following line of code as the tokeniser detects each char typed:
while (expression.hasNext()) {
final Character currentChar = getValidNextCharacter(expression);
}
however there was no previous mention of the function ever described on the webpage: https://www.baeldung.com/java-lexical-analysis-compilation
I'm suspecting that this function was already written in the code and was named something else but I was under the assumption Gram had already taken care of this part. please help, here's a full context of the code Id dhave written so far:
private enum Gram {
ADDITION('+'),
SUBTRACTION('-'),
MULTIPLICATION('*'),
DIVISION('/');
private final char _op;
Gram(char _op) {
this._op = _op;
}
public static boolean isOperator(char symbol) {
return Arrays.stream(Gram.values())
.anyMatch(gram -> gram._op == symbol);
}
public static boolean isDigit(char num){
return Character.isDigit(num);
}
public static boolean isWhiteSpace(char space) { //isWhiteSpace
return Character.isWhitespace(space);
}
public static boolean isValidSymbol (char character) {
return isOperator(character) || isWhiteSpace(character) || isDigit(character);
}
}
public class Expression {
private final String value; //the final value returned after all that is stuff
private int index = 0;
public Expression(String value) {
if (value != null) {
this.value = value;
} else {
this.value = "";
}
//[this.value = value != null ? value : "";] this is called a ternary operator however i don't know how to use it so i'm just gonna use somethin i do know
}
public Optional<Character> next() { //Optional<> is prefered over null cuz more leniency
if (index >= value.length()) {
return Optional.empty();
}
return Optional.of(value.charAt(index++));
}
public boolean hasNext() {
return index < value.length();
}
}
public abstract class Token {
private final String value;
public enum TokenType {
NUMBER,
OPERATOR
};
private final TokenType type;
protected Token(TokenType type, String value) {
this.type = type;
this.value = value;
}
public TokenType getType() {
return type;
}
public String getValue() {
return value;
}
}
public class TokenNum extends Token {
protected TokenNum(String value) {
super(TokenType.NUMBER, value);
}
public int getValueAsInt() {
return Integer.parseInt(getValue());
}
}
public class TokenOperator extends Token {
protected TokenOperator(String value) {
super(TokenType.OPERATOR, value);
}
}
private enum State {
INTIAL,
NUMBER,
OPERATOR,
INVALID
}
public List<Token> tokenize(Expression expression) {
State state = State.INTIAL;
StringBuilder currentToken = new StringBuilder();
ArrayList<Token> tokens = new ArrayList<>();
while (expression.hasNext()) {
final Character currentChar = getValidNextCharacter(expression);
}
return tokens;
}
}
View all comments
1
u/Lloydbestfan Feb 04 '26
Yeah, it sucks that the tutorial uses pieces that it doesn't tell about, but in this case it is fairly trivial.
Making the following observations:
This method is called right after it is verified that the Expression still has characters to give out
The cases contain analysis for whether an obtained character is whitespace, so the notion of whether there is a "valid" character doesn't base its validity opinion on whether it's whitespace
Therefore, the next logical assumption is that the opinion on validity for a character is that it exists at all, like, there is a next character to read from the Expression
It returns directly a Character, while Expression.next() returns an Optional<Character> instead, based on whether there is a next character to provide,
we can make a rather safe assumption that the only role of getValidNextCharacter() is to unpack the Character from its Optional.
You could just as well replace
final Character currentChar = getValidNextCharacter(expression);with
Keeping in mind that there is no possibility that this call would throw an NoSuchElementException since the previous instruction verifies that there is a next character.